AppsDBA-Sallamsetti

Wednesday, 13 August 2014

12.2 Autoconfig Issue/Fixes

you might see the below error when you run adop phase=prepare

Error:

ARNING: [CVM Error Report]
The following report lists errors encountered during CVM Phase
<filename> <return code where appropriate>
/oard/oracle/oardappl/fs1/EBSapps/appl/fnd/12.0.0/patch/115/bin/txkUpdateEBSDomain.pl -1

No. of scripts failed in CVM phase: 1

I have taken the below action plan :

1) Run autoconfig manually and verify the issues.

Below are the two scripts got failed and fixes are here as follows:

1) bisdblrp.sql

2) txkappsprf.sql

Issues & Fixes :

==========

Error 1:

WARNING: [AutoConfig Error Report]

The following report lists errors AutoConfig encountered during each

phase of its execution. Errors are grouped by directory and phase.

The report format is:

[PROFILE PHASE]

AutoConfig could not successfully execute the following scripts:

Directory: /oaph/oracle/oaphappl/fs1/FMW_Home/webtier/perl/bin/perl -I /oaph/oracle/oaphappl/fs1/FMW_Home/webtier/perl/lib/5.10.0 -I /oaph/oracle/oaphappl/fs1/FMW_Home/webtier/perl/lib/site_perl/5.10.0 -I /oaph/oracle/oaphappl/fs1/EBSapps/appl/au/12.0.0/perl -I /oaph/oracle/oaphappl/fs1/FMW_Home/webtier/ohs/mod_perl/lib/site_perl/5.10.0/x86_64-linux-thread-multi /oaph/oracle/oaphappl/fs1/inst/apps/OAPH_oacmph/admin/scripts/adexecsql.pl sqlfile=/oaph/oracle/oaphappl/fs1/inst/apps/OAPH_oacmph/admin/install

bisdblrp.sql INSTE8_PRF 1

Solution:

Drop the duplicate DB Links and then manually run the bisdblrp.sql script and the re-run autoconfig.

drop database link APPS_TO_APPS.WORLD (which is owned by obt_aa)

drop database link EDW_APPS_TO_WH.WORLD (which is owned by obt_aa)

Run the failing script (bisdblrp.sql) manually:

# cd /oaph/oracle/oaphappl/fs1/inst/apps/OAPH_oacmph/admin/install/

# sqlplus apps/<passwd>

SQL>@bisdblrp.sql

when prompted define the apps username, apps password and the DBNAME for this instance

Error 2:

adcvmlog.xml renamed to /oaph/oracle/oaphappl/fs1/inst/apps/OAPH_oacmph/admin/log/08051619/adcvmlog.xml.08051627

WARNING: [AutoConfig Error Report]

The following report lists errors AutoConfig encountered during each

phase of its execution. Errors are grouped by directory and phase.

The report format is:

[PROFILE PHASE]

AutoConfig could not successfully execute the following scripts:

txkappsprf.sql INSTE8_PRF 1

AutoConfig is exiting with status 1

AutoConfig execution completed on Tue Aug 5 16:27:42 2014

Time taken for AutoConfig execution to complete : 7 mins 56 secs

Solution:

adop phase=fs_clone -- finished sucessfully

adop phase=prepare Erroring out

I found metalink note 784254.1

but solution provided there did not worked

SELECT DISTINCT substr(object_name, 1, length(object_name)-2) q_name

FROM dba_objects

WHERE object_type = 'RULE SET'

AND object_name like 'WF%'

AND owner in ('APPLSYS', 'APPS')

AND status = 'INVALID';

still returns

WF_JAVA_ERROR

WF_JAVA_DEFERRED

WF_WS_JMS_OUT

WF_WS_SAMPLE

WF_WS_JMS_IN

WF_JMS_OUT

WF_JMS_JMS_OUT

WF_JMS_IN

Type: ODM Question

How to recreate 'APPLSYS.WF_JAVA_DEFERRED_R' subscriber?

Type: ODM Answer

1. Ensure that you have taken a backup of your system before applying the recommended solution.

2. Run the following scripts in a TEST environment first:

declare

l_wf_schema varchar2(200);

lagent sys.aq$_agent;

l_new_queue varchar2(30);

begin

l_wf_schema := wf_core.translate('WF_SCHEMA');

l_new_queue := l_wf_schema||'.WF_JAVA_DEFERRED';

lagent := sys.aq$_agent('WF_JAVA_DEFERRED',null,0);

dbms_aqadm.remove_subscriber(queue_name=>l_new_queue, subscriber=>lagent);

end;

commit;

declare

l_wf_schema varchar2(200);

lagent sys.aq$_agent;

l_new_queue varchar2(30);

begin

l_wf_schema := wf_core.translate('WF_SCHEMA');

l_new_queue := l_wf_schema||'.WF_JAVA_DEFERRED';

lagent := sys.aq$_agent('WF_JAVA_DEFERRED',null,0);

dbms_aqadm.add_subscriber(queue_name=>l_new_queue, subscriber=>lagent,rule=>'1=1');

end;

commit;

3. Once the scripts complete, confirm that the OBJECT is VALID.

You can use the following SQL to confirm:

select object_name, object_type, status from dba_objects where object_name='WF_JAVA_DEFERRED_R' ;

Sunday, 27 July 2014

Issues & fixes - 12.2

Clone issue :

below issue happens if the patch file system adcfgclone fails in between and if we are restarting, we should always clean the oraInventory directory by using the following commands:

Error :

Below Oracle Homes are already registered in the global inventory:
/oaph/oracle/oaphappl/fs2/FMW_Home/Oracle_EBS-app1
/oaph/oracle/oaphappl/fs2/FMW_Home/webtier
/oaph/oracle/oaphappl/fs2/FMW_Home/oracle_common

Exiting Cloning...

Ensure that the above Oracle Homes are unregistered from the global inventory "/oaph/oracle/oaphappl/oraInventory/ContentsXML/inventory.xml" and re-run adcfgclone.pl script

Solution :

cd $ORACLE_HOME/oui/bin

./runInstaller -silent -deinstall REMOVE_HOMES={"/oaph/oracle/oaphappl/fs2/FMW_Home/Oracle_EBS-app1"}

./runInstaller -silent -deinstall REMOVE_HOMES={"/oaph/oracle/oaphappl/fs2/FMW_Home/oracle_common"}
./runInstaller -silent -deinstall REMOVE_HOMES={"/oaph/oracle/oaphappl/fs2/FMW_Home/webtier"}

Then restart the adcfgclone.pl appsTier

==============================================

adcfgclone.pl appsTier fails after it run for 20 minutes, when it tries to create the FMW_HOME. this issue happens when the clone session closed abruptly.

Reference Document: 1682612.1

Issue :

Log file located at /local/FDEV/fs2/inst/apps/FDEV_lapdev01/admin/log/clone/ApplyAppsTier_05301531.log
- 50% completed
ERROR while running Apply...
Fri May 30 17:37:39 2014

ERROR: Failed to execute /fs02/app/FDEV/apps/fs2/EBSapps/comn/clone/bin/adclone.pl

Solution :

1. Rename or delete the file:
$ mv /var/tmp/.ovdlock.tmp /var/tmp/.ovdlock.tmp.bak

2. Restart the clone command:
$ perl adcfgclone.pl appsTier

========================================
Issue :

When you run adop phase=prepare and found below errors :

adop phase=prepare failed with the following errors

[STATEMENT] Updating prepare_status=R for session_id : 19
[UNEXPECTED]Dictionary Corrupted:
[START 2014/07/07 07:22:18] Data Dictionary Curruption Details
APPS OE_CREDIT_CHECK_ORDER_PVT V_20140516_0452 APPS OE_CREDIT_CHECK_UTIL V_20140516_0452 P Status: 5
APPS OKL_PAY_CURE_REFUNDS_PVT_W V_20140516_0452 APPS OKL_PAY_CURE_REFUNDS_PVT V_20140516_0452 P Status: 5
APPS OKL_PAY_CURE_REFUNDS_PUB_W V_20140516_0452 APPS OKL_PAY_CURE_REFUNDS_PUB V_20140516_0452 P Status: 5
APPS OKL_PAY_CURE_REFUNDS_PUB_W V_20140516_0452 APPS OKL_PAY_CURE_REFUNDS_PVT V_20140516_0452 P Status: 5
...
APPS INV_OPM_LOT_MIGRATION V_20140516_0452 APPS INV_LOT_API_PUB V_20140516_0452 TS mismatch: 12-JUN-14 16:50:54 29-JUN-14 14:42:12
...
APPS INV_LOT_ATTR_PUB V_20140516_0452 APPS INV_LOT_API_PUB V_20140516_0452 TS mismatch: 12-JUN-14 16:50:54 29-JUN-14 14:42:12
APPS INV_LOT_ATTR_PUB V_20140516_0452 APPS MTL_LOT_UOM_CONV_PVT V_20140516_0452 P Status: 5
APPS CSD_SC_DOMAINS_PVT V_20140516_0452 APPS CSD_GEN_UTILITY_PVT V_20140516_0452 P Status: 5

Solution :

Reference: R12.2 Dictionary Corrupted While Running adop - "TS Mismatch" Error (Doc ID 1576086.1)

login to Database server :

1) sqlplus "/ as sysdba"
shutdown immediate
startup upgrade

2) @utlirp.sql

3) Bounce the database again and open it in normal mode.

From the same sqlplus session
shutdown immediate
startup
4) @?/rdbms/admin/utlrp.sql

5) re-run adop.

==========================================

Issue :

ERROR at line 1:
ORA-04063: package body "SYS.DBMS_REGISTRY_SYS" has errors
ORA-06508: PL/SQL: could not find program unit being called:
"SYS.DBMS_REGISTRY_SYS"
ORA-06512: at line 1

Solution:

How To Validate an Invalid SYS.DBMS_REGISTRY_SYS Package (Doc ID 438821.1)

SQL> alter trigger SYSTEM.DROPLOG_TRIGGER DISABLE;

========================================
Error :

[apploaph@oacmph scripts]$ sqlplus apps/rwnb1adr12

SQL*Plus: Release 10.1.0.5.0 - Production on Mon Jul 7 23:18:22 2014

Copyright (c) 1982, 2005, Oracle. All rights reserved.

ERROR:
ORA-04045: errors during recompilation/revalidation of SYSTEM.EBS_LOGON
ORA-01031: insufficient privileges

Enter user-name:

Solution:

SQL> select trigger_name,owner,status from dba_triggers where trigger_name='EBS_LOGON';

TRIGGER_NAME OWNER STATUS
------------------------------ ------------------------------ --------
EBS_LOGON SYSTEM ENABLED

SQL> alter trigger SYSTEM.EBS_LOGON DISABLE;

Trigger altered.

================================================

Issue :

*** Log File = /oaph/oracle/oaphappl/fs_ne/EBSapps/log/adop/19/prepare_20140707_233754/OAPH_oacmph/TXK_SANITY_filesystemcheck_Mon_Jul_7_23_39_23_2014/txkADOPPreparePhaseSanityCheck_Mon_Jul_7_23_39_23_2014.log

[ERROR] Minimum file system space required 25 GB while you have 17.982421875 GB

[UNEXPECTED]Error occurred while executing "perl /oaph/oracle/oaphappl/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSanityCheck.pl -contextfile=/oaph/oracle/oaphappl/fs1/inst/apps/OAPH_oacmph/appl/admin/OAPH_oacmph.xml -patchcontextfile=/oaph/oracle/oaphappl/fs2/inst/apps/OAPH_oacmph/appl/admin/OAPH_oacmph.xml -promptmsg=hide -mode=update -console=off -sessionid=19 -timestamp=20140707_233754 -outdir=/oaph/oracle/oaphappl/fs_ne/EBSapps/log/adop/19/prepare_20140707_233754/OAPH_oacmph -action=filesystemcheck"
[UNEXPECTED]Error while runFSSanityValidateConfig sub-routine is called
[ERROR] Prepare phase completed with errors or warnings. Please check logfiles.
Log file: /oaph/oracle/oaphappl/fs_ne/EBSapps/log/adop/19/adop_20140707_233754.log

adop exiting with status = 1 (Fail)
You have new mail in /var/spool/mail/applo

Solution :

Make sure the free space is around 40 GB

=================================================
Issue :

Multiple context files exist in application side . you can view them through OAM.

because of this, you cannot add/modify customizations on your environment file.

Solution:

Remove the inconsistent row from FND_OAM_CONTEXT_FILES

STEPS:

1. Run the following script to clear out FND_OAM_CONTEXT_FILES of all entries. This will also
clear other tables like FND_NODES:

EXEC FND_CONC_CLONE.SETUP_CLEAN;
COMMIT;
EXIT;

2. On the database Node, run autoconfig:

$ORACLE_HOME/appsutil/scripts/<context_name>/adautocfg.sh

3. On All APPS nodes run autoconfig:

$ADMIN_SCRIPTS_HOME/adautocfg.sh

4. Retest the issue.

NOTE: Step 1 will clean out all information from some FND tables, so it is important that
autoconfig is run on ALL nodes in this environment. Must run on DB nodes first.

OR

1> Backup the table: create table applsys.fnd_oam_context_files_bak as select * from
applsys.fnd_oam_context_files;

2> Truncate the table: truncate table applsys.fnd_oam_context_files

3> Run Autoconfig on all nodes to reload the context file information to the database.

4> Re-Run Autoconfig to confirm that the issue does not replicate after the first run.

5> Check the FND_OAM_CONTEXT_FILES table to ensure the data was re-populated.

Create a Custom Parameter Fails With - Customization Is Not Allowed On Different Versions of Configuration Files [ID 762590.1]
CUSTOMIZATIONS NOT ALLOWED ON DIFFERENT VERSIONS OF CONFIGURATION FILES [ID 351969.1]

Monday, 30 June 2014

ADOP Basic Questions & Answers

What is the concept of ADOP in R12?

Online patching is the most important new feature in Oracle E-Business Suite Release 12.2. It is the ability to patch a running system without having to take the system down for a significant period of time while the patches are applied.

‘adop’ is the utility you use to apply application patches in R12.2

Note: Once you enable ADOP on R12.2 system you cannot use adpatch utility

What is the patch directory in R12.2

In R12.2 there is a new directory location environment variable called $PATCH_TOP which points to $NE_BASE/EBSapps/patch
$NE_BASE points to <Non-Editioned-filesystem-directory> or you can use $APPL_TOP_NE/..

Download the patch into the patch top directory and unzip it. This is the default location where the adop will look for patch files.

If you want to use other directory to maintain the patches   then you need to use the below parameter while applying the ADOP.

i.e. adop phase=apply patchtop=/u01/R122_Patches/<Patch_number>

From which location to ADOP command to execute?

We always run adop utility from the run edition file system. The adop process automatically sets its environment correctly, regardless of the edition it is run from.

RUN

How to findout on which environment we logged in?

When you set the environment file as soon as you logged in as “applmgr” it prompts RUN/ PATCH Files system with full location and also prompts select the required files system

[root@R12test ~]# su - appltest

   E-Business Suite Environment Information

RUN File System           : /u01/oracle/testappl/fs1/EBSapps/appl

PATCH File System         : /u01/oracle/testappl/fs2/EBSapps/appl

Non-Editioned File System : /u01/oracle/testappl/fs_ne

DB Host: oadbph.mfc.corp.mckee.com Service/SID: TEST

E-Business Suite Environment Setting

- Enter [R/r] for sourcing Run File System Environment file, or

- Enter [P/p] for sourcing Patch File System Environment file, or

- Enter anything else to exit
Please choose the environment file you wish to source [R/P]: R

What happens when ADOP command is executed ?

adop will perform all the tasks required to apply the patch:

Ø Reads patch metadata to determine patch dependencies and requirements

Ø Attempt to recover previously failed patching session (if any)

Ø Reads and validate the patch/product driver files

Ø Compares version numbers of existing files against the patch files and Backs up all existing files that will be changed by the patch

Ø Copies files

Ø Archive files in libraries

Ø Relinks executables, Generates forms, reports, messages, graphics, and Java archive (JAR) files

Ø Compiles JSP files and invalid database objects. Updates database objects

Ø Runs AutoConfig if required

Ø Saves patch information to the database

Can we merge applications patches apply in R12.2?

Yes, but the application of merge patches are different as ADOP utility itself has a feature of merging multiple patches.

In earlier EBS releases, the AD Merge Patch tool was used to merge multiple patches into a single patch, so that the common tasks only needed to be performed once.

By default, adop will automatically merge all patches specified with the ‘patches’ parameter.

You can still use earlier AD Merge Patch tool but you should disable adop’s merging of patches, by adding to the adop command line > ‘merge=no’

Eg: adop phase=apply patches= 18471329, 18132674….

Note: You can give any number of patches and with comma separated value and apply them in a single command shown above.

Friday, 27 June 2014

R12.2_ Online Patching basics

ADOP over view:

=============

In Release 12.2.0 and higher, patching is performed by running the new adop (AD Online Patching) utility. We must use adop utility instead of adpatch utility that was provided in previous releases. Applications EBS 12.2 introduces online patching using ADOP utility. Technology patches are applied to the patch file system then ADOP is used to propagate the change into run within the context of a patching cycle.

To start Online Patching, we need to use:

$ adop phase=prepare

To Apply Patches, we need to use:

$ adop phase=apply patches=9876789,8768904

ADOP Features:

============

To obtain help about the basics of adop operation, use:

adop –help

This will display the following text:

Applications DBA Online Patching Tool (adop)

Usage: adop [phase=<phase,phase,...>] [patches=<patch#,patch#,...>]

[<parameter>=<value> ...] [input_file=<filename>]

Enter adop -examples for a detailed list of parameters and their usage.

The phase parameter specifies the parts (phases) of the online patching cycle to be executed.

The five standard phases are executed in the order shown below.

Standard phases:

prepare - Prepare the instance for patch application.

apply - Apply patches (to the patch edition).

finalize - Ready the instance for cutover.

cutover - Make the patch edition the new run edition.

cleanup - Drop obsolete objects and data from old editions.

There are also three special phases, for use when needed.

Special phases:

abort - Abort the current patching cycle.

actualize_all - Create new copies of all code objects in the patch edition.

fs_clone - Copy the run file system to the patch file system.

Phase-specific parameters control operation of a particular phase:

Apply parameters:

Patches=<patch#>[,<patch#>...]

A single patch or comma-separated list of patch to apply.

This parameter is required when executing the apply phase.

For language patches, you must also specify the driver file:

patches=10987654_AR:10987654.drv,10987654_KO:10124645.drv

restart=(yes|no) [default: no]

Resume a failed apply action where processing left off.

abandon=(yes|no) [default: no]

Re-apply a failed patch from the beginning.

Finalize parameters:

finalize_mode=(full|quick) [default: quick]

Quick mode will provide the shortest execution time, by skipping non-essential additional actions.

Full mode performs additional actions such as gathering statistics, which may improve performance after cutover.

Cutover parameters:

mtrestart=(yes|no) [default: yes].

Specifies whether to restart application tier servers after cutover. Leave at default unless performing manual steps during downtime.

Cleanup parameters:

cleanup_mode=(full|quick) [default: quick]

Quick mode provides the shortest execution time, by

skipping non-essential additional actions.

Full mode performs additional processing to remove all unused code, data, and old editions. May take a long time.

General parameters apply to all phases:

workers=<number> [default: computed]

Number of parallel workers used to execute tasks.

Default value is computed principally according to number of available CPU cores.

input_file=<file_name>

As well as being entered directly on the command line, adop parameters can be specified in a text file, with

one <parameter>=<value> on each line of the file.

Examples:

phase=prepare,apply,finalize,cutover,cleanup

patches=123456

workers=4

Command line parameters override input file parameters.

[default: event]

Controls the level of diagnostic log detail displayed.

allnodes=(yes|no) [default: yes]

Specifies whether actions should be executed on all application tier nodes of a multi-node system.

action=(db|nodb) [default: db]

Specifies whether to execute database actions (as well as file system actions).

-status [<session_id>]

Display status of the latest adop session, or a specified session.

-help

This help screen.

-examples

Source copied from other sites for my reference.