************************************************************************** ** All Copyright reserved by Vishal Jagtap. Email : [email protected] ** ************************************************************************** >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> (1) Setting ContinuityCheck to true or false set aal2pathvcctp=bsiteid ContinuityCheck true/false get swithcmodule=es-1 //active module get aal2pathvcctp=b....a1 vcl >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> (2) The formulas used by moshell to calculate the KPIs under pmr command are located in /moshell/commonjars/ pm/*Formulae* >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> (3) To check the actual load on the RNC use the following command. $ capi tot CPU load report --------------Integration interval: 100000 microseconds Load (%) 2 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> (4) To check the load on each MP and BP. It also displays the temp of each board and load on each board. $ cabl >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> (5) to get the ram usage on each board on RNC $ cabm >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> (5.1) To get the log of abnormal restarts for MP/BP use the following commands Rnc99> cabra >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> (6) To check the state of the devices in RNC use the following command Rnc99> std >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> (7)RBS- vii- use this command to check the clients registered with GPB. You can see whether the GPB is busy or not from this command 3710_YishunSt71Blk715> vii 081120-00:06:12 rbs3710 7.0t RBS_NODE_MODEL_J_5_17 stopfile=/tmp/3699 Trying password from ipdatabase file: /home/ericsson/tools/moshell/sitefiles/ipdatabase... $ vii ----- VII SERVER INFO ----registered clients: client FAULT LOAD_START NO_POWER BOOTTEST MISSING_RESOURCE BOARD_LOCKED BOARD_BUSY SHUTDOWN ---------- ----- ---------- -------- -------- ---------------- --------------------- -------0x000102d7 NO NO NO NO NO NO YES NO LED GREEN RED YELLOW $ State CLS_LEDI_ON CLS_LEDI_OFF CLS_LEDI_FLASHING_16HZ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (9) RNC- to check the no of PS and CS calls happening at the RNC for the given moment. Rnc01> lhsh 001700 ranap status >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (10) to understand what "hc" command would do iinternally, please check the C:\cygwin\home\evisjag\moshell\commonjars\scripts\hc_datacollection.mos file >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (11)Delete license file: Application on P5. Havent checked on p6 or P7. 1. 2. 3. 4. 5. 6. login to rbs using moshell check the current alarms and calls on the rbs go to cd /c/license remove all the license file do a cold restart using �acc 0 restart� y / 2 (for cold restart) relogin to rbs and check license key by using command �license key� Note that only the file that will be completly delete is licensekeys.xml file. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (12.1) To get the invetory list for the TX and RAX boards you need to telnet to OSS RC and then use the following command. #Create a Selection file named all_RBS_signtel_xxxxx.sel in the /var/opt/ericsson/nms_smo_srv/smo_file_store/Selection/ directory using OSSRC. 1) Telnet to OSS-RC 10.252.75.35. Use any account 2) Go to home directory of the user 3) Use command "smocpp" 4) Then use command "findpiu -nefile 270910_rnsAll_rbs.sel -name RAX* " - for RAX board inventory list. 5) Then use command "findpiu -nefile 270910_rnsAll_rbs.sel -name TX* " - for TX board inventory list. 6) Log the session to get the inventory list using the Secure CRT options. **** The important thing here is that the smocpp tool uses parent directory as "/var/opt/ericsson/nms_smo_srv/smo_file_store/Selection/" hence the RBS list should be in the parent directory and not in the local directory. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (12.2) To get Channel Element Utilization report. You can import the script in your home directory e.g. /home/evisjag/ After that try to run the command /opt/ericsson/nms_umts_cms_lib_com/bin/run_moscript <space> <the script file name with extension> >/dev/null Ex: /opt/ericsson/nms_umts_cms_lib_com/bin/run_moscript PKGBE.py >/dev/null As .csv file containing the required information will be generated which will have all the required information. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (12) RNC upgrade health check related commands lt all hdinfo rbs hdreadtest hdstat vols readclock mmu pglist >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (13) if you need to check what happened on the RBS or RNC so far, who loged in and what they did use the following command $lgaevsyoulmr >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (14) Following is some information related to RNC design Understanding RNC system Hard drives on boards 1) when you log into RNC then you log into the main MP (Fault tolerant core) board which is generally 001000. 2) The /c drive is actually only on the board 001000. It acts as an network drive and you can access it from any other board. GPB. 3) the /c partition is mirrored over to the 001100 board i.e passive core 4) the /d drive in 001000 board contains the CV configuration, loadmodules, system files etc. 5) the drive for other GPBs are network drives and should be accessed as /p001100 for drive on GPB11. 6) boards such as TUB, SCB or SPB does not have any hard drive (CF) 7) If you logged into RNC and then do "te log read" then the logs are from the main MP. 8) If LinkHandler ex; # # # you need to check the logs from any other board then you need to use SHell commands i.e. lhsh lhsh 001700 te log read; lhsh 0017000 ranap status ; lhsh 002600 vii;llog; etc. 9) If you need to check whether there was any abnormal restart on any particular board then use the commans # lhsh 002600 llog -l 2 (2 specifies last 2 restarts only) 10) If you want to check which one of the board 10 or 11 is active you can do so using #mirror stat (on 001000 boards) you can also do (#lhsh 001100 mirror stat) to check the status of the board. This only works for main MP/ passive MP. 11) if you want to check the active program list running on a board then you can do following #lhsh 001100 sma -all 12) If you wish to check the program list on a specific board then you can do following #lhsh 001100 pglist 13) If you wish to check the processes running on a specific board then you can do following #lhsh 001100 ps 14) In CPP software commes in loadmodules (LMs). LM is a single file that implements certain functionality. LM runs on the boards in Cello. Once you load and start the LM, you have PROGRAMS. Each program then consist of PROCESSES. 15) A CV contains following things db.dat - dump of database. (dont know what exactly is in there.) ARMAMENT - A file containing list of load modules to be loaded on the boards once system starts. LLP.LMID - A file contains list of load modules that contains load list provider (LLP). LLP knows which loadmodules to be loaded on each board and in which order. When individual board comes up they ask LLP for "shopping list". After that they turn to LMP (load module provider) to get load modules. Each MP can be LMP and LLP is last resort if there is no other LMP. attributes - who created? when? reason etc. ok - to make sure CV is not corrupted. Just cotains a file named ok with "ok" inside. 16) Boot up process a) From PROM of main MP get the boot infor and initialize CPU and some other chips b) boot up the OS to select either BASIC or backup mode (lets say OS choos BASIC mode) c) load basic modules (there are 2 3 basic modules to select CV and start up system) d) /c is /c here e) do /c/cv.ptr exist. This will point to latest CV (assume it exists) f) get CV name from /d/configuration/cv/xxxx/. Check if ok file is there (to confirm CV is not corrupt) g) load and start LLP.LMID h) LLP reads ARMAMENT and load load modules in specific board to get system up far enough so that rest can be done using db.dat i) load database with containts of db.dat ( this contains all other load modules and their config data) j) All boards then ask LLP for "shopping list" of required modules k) retrieve load modules from load module provide (LMP) . Each MP can be LMP as it shorten up the boot time. l) start the load modules m) all MP, BP started and OMS initialized. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (15)Concept of RNC module One RNC module consist of a GPB, SPMs from SPB boards and terminated sites through ET boards. The allocation of the SPM devices is dynamically done and is handled by the main MP. The SPB contains 3 types of processors a) PDR devices(packet data router) - for GTP-U towards CN b) CC devices (common channel) - used for configures radio channels e.g. FACH,CCCH c) DC devices (dedicated channel)- used for dynamic radio channel e.g. DCH For a given site we need to know 1) RNC module where the site is located. for this use command #get iublink rncmodule. For specific site do #get iub_3999 rncmodule ex: Rnc99> get iub_3999 rncmodule 090213-17:01:35 10.251.88.52 7.0t RNC_NODE_MODEL_J_3_25 stopfile=/tmp/21994 =================================================================================== ============================== MO Attribute Value =================================================================================== ============================== IubLink=Iub_3999_OLP_RBS3202_2 rncModuleRef RncModule=1 =================================================================================== ============================== Total: 1 MOs 2) The location of RNC modules i.e. which GPB which subrack.To get this just do #bp mod10 (for the above example) ex: Rnc99> bp mod1 090213-17:02:10 10.251.88.52 7.0t RNC_NODE_MODEL_J_3_25 stopfile=/tmp/21994 ================================================ Board BoardType Swallocation/DevType Module ================================================ 001400 GPB53 GPB_Module 1 3) on which ET board the site is terminated. You can do this by #get aal2path.*3999 ex: Rnc99> get aal2path.*3999 090213-16:59:49 10.251.88.52 7.0t RNC_NODE_MODEL_J_3_25 stopfile=/tmp/21994 =================================================================================== ============================== 342 TransportNetwork=1,Aal2PathVccTp=b3999a3 =================================================================================== ============================== Aal2PathVccTpId b3999a3 aal2PathId 3 aal2PathOwner true aal2QoSAvailableProfiles 12 (CLASS_C_D) aal2QoSProfileId Aal2QosProfile=ad8 administrativeState 1 (UNLOCKED) alarmReport 2 (ALARM_LOC) availabilityStatus 0 (NO_STATUS) continuityCheck true counterActivation false counterMode 6 (PM_MODE_FPM_BR) nomPmBlocksize 1024 (NOM_PM_BLKSIZE_2_10) operationalState 1 (ENABLED) remoteBlockingState 1 (REMOTELY_UNBLOCKED) reservedBy Aal2Sp=1,Aal2Ap=b3999,Aal2PathDistributionUnit=1 timerCu 10 userLabel AAL2TP_3999_RNC00_IUBAP_3 vclTpId AtmPort=ES1-261,VplTp=vp6,VpcTp=1,VclTp=1 =================================================================================== ============================== It shows that the site is terminated from RBS to RNC in ES1. For this site we can see that Iub termination is on different subrack while the RNC module is on different subrack. This means that for the ISL will be used for every call from UE to CN. The capacity of ISL is 300 Mbps. There is a possibility that if there are many sites in the RNC which are configures in this way then it might created problems due to ISL congestion. This kind of configuration should be avoided. To check this inconsistency report use the command #lki In P5 this command gives you the information about the subrack in which sites are terminated and lot more. This command is no more valid in IPRAN. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (16) To see what scanners are currently active in a RNC use pgets Ex: pgets all vcltp pgets . vpltp pmom is used to get the counters for related mos. ex: pmom vcltp pmom vpltp >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (17) To check the value of the counters use pmx commands. Ex: If you need to check Eul RAB establishment Success rate then use following command at RNC pmx -m 3 cell=5401 pmNoRabEstablishAttemptPacketInteractiveEul pmx -m 3 cell=5401 pmNoRabEstablishSuccessPacketInteractiveEul >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (18) to get the celldata for a given site on rnc modul8 or to check the hanging resources $ cedhr # See hanging resources $lh mod8 celldata -cid 38453 $lh mod8 hwm -cid 38133 print grp $lh mod8 hwm -cid 38133 print rbs $lh mod8 hwm -cid 38133 print cell $lh mod8 codetree -cid 381333 You can see HSDPA/EUL user count in the cell >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (19) To check the ammount of credit left in a cell use the command $lh mod8 hwm -cid 38173 print grp >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (20) To get the pmd files for the crashed devices dump list -a | sort +2 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (21) Decode pmd files use zpm zpm 0x...pmd to get the decoded pmd file for online decoding i.e within RNC itself use following command $ dump error -i 0x...pmd ------this gives a short info on pmd crash $ dump -i 0x...pmd ------this decodes pmd file completely >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (22) How to update DNS server. Telnet to 10.252.75.13 root/singtel123. 287 cd /var/named 289 vi utran.3g.singtel.com //Edit this file corrosponding site 292 vi 104.10.in-addr.arpa //Edit this file corrosponding site 294 vi 146.251.10.in-addr.arpa //Edit this file corrosponding site then restart ... 298 svcadm restart svc:/ericsson/cominf/dns/server:default >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (23) RNC babysitting commands for IuCS and IuR monitoring -----------------------------------Normal SMS contains -------------------1. RNC load - pmr option 6 - choose the highest load 2. RRC attempts - pmr option 1 ( in number) 3. RRC Success - pmr option 1 ( in %) 4. No of CS calls - from "lhsh 001700 ranap status" 5. No of PS calls - from "lhsh 001700 ranap status" ----6. RNC iur drop rate - pmr option 31 (for checking only) Check signaling (CP) --------------------1. lst mtp3bsl 2. lst gs 3. lst ranap 4. lst rnsap Check user plane (UP) --------------------1. lst aal2path.*gc ==> check UP towards MGW01/03 Check boards ------------1. lst ms-6-2 2. lst ms-7-2 Check counters --------------lpr aal2ap.*g3 ==> get proxy id of mo pget TransportNetwork=1,Aal2Sp=1,Aal2Ap=g3 Ex: Aal2Sp=1,Aal2Ap=g3 Aal2Sp=1,Aal2Ap=g3 Aal2Sp=1,Aal2Ap=g3 Aal2Sp=1,Aal2Ap=g3 pmSuccInConnsRemoteQosClassA 0 Aal2Sp=1,Aal2Ap=g3 pmSuccInConnsRemoteQosClassB 0 Aal2Sp=1,Aal2Ap=g3 pmSuccInConnsRemoteQosClassC 0 Aal2Sp=1,Aal2Ap=g3 pmSuccInConnsRemoteQosClassD 0 Aal2Sp=1,Aal2Ap=g3 pmSuccOutConnsRemoteQosClassA 4926 Aal2Sp=1,Aal2Ap=g3 pmSuccOutConnsRemoteQosClassB 13 Aal2Sp=1,Aal2Ap=g3 pmSuccOutConnsRemoteQosClassC 0 Aal2Sp=1,Aal2Ap=g3 pmSuccOutConnsRemoteQosClassD 0 Aal2Sp=1,Aal2Ap=g3 Aal2Sp=1,Aal2Ap=g3 Aal2Sp=1,Aal2Ap=g3 pmUnSuccInConnsLocalQosClassA 0 Aal2Sp=1,Aal2Ap=g3 pmUnSuccInConnsLocalQosClassB 0 Aal2Sp=1,Aal2Ap=g3 pmUnSuccInConnsLocalQosClassC 0 Aal2Sp=1,Aal2Ap=g3 pmUnSuccInConnsLocalQosClassD 0 Aal2Sp=1,Aal2Ap=g3 pmUnSuccInConnsRemoteQosClassA 0 Aal2Sp=1,Aal2Ap=g3 pmUnSuccInConnsRemoteQosClassB 0 Aal2Sp=1,Aal2Ap=g3 pmUnSuccInConnsRemoteQosClassC 0 Aal2Sp=1,Aal2Ap=g3 pmUnSuccInConnsRemoteQosClassD 0 Aal2Sp=1,Aal2Ap=g3 pmUnSuccOutConnsLocalQosClassA 0 Aal2Sp=1,Aal2Ap=g3 pmExisOrigConns pmExisTermConns pmExisTransConns 1293 0 0 pmUnRecMessages pmUnRecParams 0 0 pmUnSuccOutConnsLocalQosClassB 0 Aal2Sp=1,Aal2Ap=g3 pmUnSuccOutConnsLocalQosClassC 0 Aal2Sp=1,Aal2Ap=g3 pmUnSuccOutConnsLocalQosClassD 0 Aal2Sp=1,Aal2Ap=g3 pmUnSuccOutConnsRemoteQosClassA 0 Aal2Sp=1,Aal2Ap=g3 pmUnSuccOutConnsRemoteQosClassB 0 Aal2Sp=1,Aal2Ap=g3 pmUnSuccOutConnsRemoteQosClassC 0 Aal2Sp=1,Aal2Ap=g3 pmUnSuccOutConnsRemoteQosClassD 0 Signaling through MGW ----------------------lst aal5tpvcctp.*g ==> signanling towards MGW for establishing IuCS UP i.e to support QAAL2 signaling and layers below it lst aal5tpvcctp.*cs ==> signaling towards core network (IuCS CP) lst aal5tpvcctp.*r ==>signaling towards other RNCs(RNSAP CP) Up through MGW ---------------lst aal2path.*=g ==> UP towards MGWs (IuCS UP) lst aal2path.*=r ==> UP towards other RNCs (RNSAP UP) For IuR the RNSAP UP is cross connected in MGW. For IuR RNSAP CP signaling - it goes through MGW to the other RNC. It is defined by aal5tpvcctp.*r links. The IuR destination point codes are directly towards other RNC. Summary: --------All IuCS, IuR links are from RNC goes through MGW. Destination point code is for SS7 networks can be found only in CPs in Mtp3bSrs. (AAL5 is used for CP) In a given system there Endpoint Addresses) aal2ap is for QAAL2 and aal2routingcase. For UP aal2path carries atm address(destination is single AAL2SP which has its own a2ea (AAL type 2 Service it is used for establishing the UP. aal2ap is used by traffic. aal2routingcase routes the traffic to destination number). IuCSoATM -------IuCS CP - from RNC03 goes to MGW03/05/01 then towards MSC. Terminates at MSC core. - As it is CP it uses AAL5 and represented by aal5tpvcctp=cs IuCS UP - It is from RNC03 to MGW03/05/01. Terminated at MGW. - To establish IuCS UP RNC and MGW needs to do signaling hence need to use AAL5. It is represnted by aal5tpvcctp=g (uses destpoint code to MGW) - For Qaal2 system uses aal2AP(access point) and is reserved by aal2routing case. towards routing carried Child of AAL2AP is aal2distribution unit which reserves all aal2path MGW for IuCS-UP. aal2routingcase has numberDirection which is A2EA . This defines how the is performed(1hop) Once this signaling beteween RNC and MGW is established then traffic is over AAL2 as per aal2routingcase. Additional Babysitting useful commdands -----------------------------------------#COMMANDS APPLICABLE TO RNC AND RXI: #verify Upgrade Package executing is the correct one cvcu #check HW status and look for spontaneous restarts cab cabr #look for new alarms al #Peer Disk is Connected and Status Valid mirror sta vols hdstat lhsh 001100 hdstat #processor load is normal pget pluginunit pmProcessorLoad #no aal2 rejections pget aal2ap pmunsucc #no new events or alarms related to the update/upgrade lgaevusm #COMMANDS APPLICABLE ONLY TO RNC: #number of ranap connections pending lhsh 001700 ranap status #cell availability is normal str #no new devices are disabled std #check al least options 4 and 45 #for optin 4 pay special attention to: # # HSMinperDrop # HsAccess # PSAccess # PSDrop # # # # # # # # # pmr PSRabSucc PSRrcSucc RrcSuc SpchAccess SpchDrop SpchRabSuc SpchRrcSuc #all types of traffic are running uer #look for new disable MO�s st . disable ------------------------------------------------------------------------------------------------------------------------------IuRoATM IuR CP - goes through MGW to the other RNC. The destination point code is of the other RNC. - it is carried over aal5 and named under aal5tpvcctp.*r IuR UP - goes through MGW to the other RNC. MGW has to cross connect the aal2paths towards both RNCs. - represnted by aal2path.*r ------------------------------------------------------------------------------------------------------------------------------IuPSoIP CP and UP goes through SGSN. CP - RANAP-PS is running on SGSN. UP - goes on with GTP-U application part. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (24) Linux - UNIX related (A)Screen Command a) Use command "$screen" to starts a new screen in the background and run mobatch b) To detach from this screen--> "Ctr a d " c) To reattach to the screen use "$screen -list" => 5154.pts-25.ompc02 (Detached) and then to connect to this screen do this - screen - r 5154.pts-25.ompc02 // connect back again to the screen. d) To kill the screens do the following 1. screen -list 5154.pts-25.ompc02 (Detached) 2. kill -9 5154 3. screen -wipe >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (25) Carrier Frequencies used by SingTel f1 - 10664 f2 -10688 9714 9738 (26) How to correctly configure HSDPA and EUL for given RBS Hardware Required for HSDPA and EUL config -------------------------------------------1> RAX - RAX boards from hardware generation R2 onwards support HSDPA and EUL. Exception : RAX boards of hardware generation R1 (RAX R1) are not HSDPA or EUL compatible. Any RAX R1 must be removed from the RBS. 2> TXHS - TX6HS-0x Excpetion : Normal TX boards. Ex - TX, TX6-01 etc 3> ET - ET-MC1, ET-M3, ET-M4 and ET-MC41s, Exchange Terminal (ET) boards are all recommended for HSDPA and EUL. Exception : ET-MC1 as it does not have IMA SW Reuirement -------------Should be above P5 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (27) How to replace a GPB board Here is the process for replacing same/similar type of boards 1. Lock the board 2.Directly connect to the board using serial cable. 3. reload --, vols, fomathd /d, formathd /c2 4. remove the board 5. Insert this board into the RNC..wait until board is loaded (can check using vii or the green LED should be lit up) 6. Unlock the board. (no need to restart the board) Serial log for formating the board��� reload--: command not found $ reload -OSE Delta PowerPC/OSE4.4.3 2.8.3 Stop due to remain in basic/backup set Welcome to OSE Shell OSE4.4.1. $ lhsh fo vols volume total used free frw device unit / 0K 0K 100% 0K 100% FR/null 0K 0K 100% 0K 100% FRW /d 683M 373M 55% 310M 45% FRW idedd 0 <4-1399999> /c2 1317M 1M 0% 1316M 100% FRW idedd 0 <1400000-4098527> $ formathd /d All data on volume will be destroyed. Continue? y/n (n):y Formating Hard Disk... $ formas thd /c2 All data on volume will be destroyed. Continue? y/n (n):y Formating Hard Disk... $ vols volume total used free frw device unit / 0K 0K 100% 0K 100% FR/null 0K 0K 100% 0K 100% FRW /d 683M 0M 0% 682M 100% FRW idedd 0 <4-1399999> /c2 1317M 1M 0% 1316M 100% FRW idedd 0 <1400000-4098527> $ #If you are replacing a GPB with another GPB then just insert the new GPB with same or higher HW revision will do There is a difference between reload -- and reload. reload -- will take the board in backup mode. In backup mode you can see drive /d and /c2 drive. reload will reboot the node in basic mode. You will not use reload -- unless its a new RNC or RBS configuration. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (28) This is used for example when we have problems to open Emas. Processes list listloaded This is used for example when we have problems to open Emas. We must check on the list for (at the program handle 14) the follow process: Load module id: CXC1320784_R16V02 Load module path: @(#)/vobs/cello/osa/java_dm/jvm_lm jvm.ppc750 ppc750 tcssapi_icp Load module version: @(#)CXC1320784 R16V02 2006-02-22 12:04:55 IST ac10330d0b8ec28f-217c Program handle: 14 Entry point: 03694a84 Start section 0: 03680068 Size: 002ba0e8 Start section 1: 039b0000 Size: 00000000 Start section 2: 039b0000 Size: 00049958 That process provides to run EMAS. So if we have a problem we can make the restart of this process: pgkill 14 Usually when the process to will be recharged the connection with EMAS is up. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (29) How to use diff in moshell to check parameter setting To compare the current parameter setting for all parameters in the RNC with the default parameter do the following 1. Get the kget for the rnc ex: kget_rnc01.txt 2. go to offline moshell ex: $moshell 3. Use following command OFFLINE> diff kget_rnc01.txt You will be able to compare the parameter setting. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (30) In moshell there are shortcut commands defined. You can check them using command $ alias Ex : 5144_Telecom_Equipment> alias err : te log read |grep ERROR: ll : llog llc : llog -c lll : llog -l ombat : run /opt/utran/moshell/commonjars/scripts/ombat/ombat.mos teall : te e check error enter return info trace1 trace2 trace3 trace4 trace5 trace6 trace7 trace8 trace9 state_change bus_send bus_receive rec_sig send_sig param interface object user1 user2 user3 user4 tec : te log clear ter : te log read >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (30.1) How to enable logging on the module. To check what are the traces enabled for the given module Rnc99> lh mod1 te status NBAP* 090909-18:04:03 rnc99 7.1n RNC_NODE_MODEL_K_9_95 stopfile=/tmp/16910 ==================================================================== $ lhsh 001400 te status NBAP* 0014: pid name enabled groups 0014: - NBAP_ASN check error info interface object 0014: - NBAP_ASN_CELL check error info interface object 0014: - NBAP_ASN_PEEK check error info interface object 0014: - NBAP_ASN check error info interface object 0014: - NBAP_ASN_CELL check error info interface object 0014: - NBAP_ASN_PEEK check error info interface object 0014: - NBAP_ASN check error info interface object 0014: - NBAP_ASN_CELL check error info interface object 0014: - NBAP_ASN_PEEK check error info interface object 0014: - NBAP_ASN check error info interface object 0014: 0014: - NBAP_ASN_CELL check error info interface object - NBAP_ASN_PEEK check error info interface object This means that there are no NBAP traces enabled on this module as enabled groups column shows " check error info interface object". to enable traces $ lhsh 001400 te enable (group) NBAP* (group) Can be one of: check, error, enter, return, info, trace1 to trace9, state_change, bus_send, bus_receive, rec_sig, send_sig, param,user1 to user4. You can also specify all to enable all trace groups for a specific process. You cannot use all together with the wildcard on the process name. This is to avoid enabling too many trace groups on too many processes. Ex: $ lhsh 001400 te enable bus_send bus_receive NBAP* $ lhsh 001400 te enable bus_send bus_receive RRC* To disable enables traces on a module use following $ lhsh 001400 te default NBAP* $ lhsh 001400 te default RRC* >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (31) How to enable UEH_EXCEPTION capturing in ITK. 1. Check the previous process ps -ef | grep monitor [itk@sgp-ominf-itk Rnc04]$ ps -ef | grep monitor itk 14154 12198 0 17:55 pts/16 00:00:00 grep monitor itk 15217 1 0 Nov05 ? 00:07:06 /opt/utran/moshell/monitor6054 rnc01.utran.3g.singtel.com 33079 itk 16009 1 0 Nov05 ? 00:04:35 /opt/utran/moshell/monitor6054 rnc02.utran.3g.singtel.com 33079 itk 16314 1 0 Nov05 ? 00:09:57 /opt/utran/moshell/monitor6054 rnc03.utran.3g.singtel.com 33079 itk 16649 1 0 Nov05 ? 00:10:09 /opt/utran/moshell/monitor6054 rnc04.utran.3g.singtel.com 33079 itk 16955 1 0 Nov05 ? 00:08:55 /opt/utran/moshell/monitor6054 rnc05.utran.3g.singtel.com 33079 itk 17260 1 0 Nov05 ? 00:08:37 /opt/utran/moshell/monitor6054 rnc06.utran.3g.singtel.com 33079 itk 19182 1 0 Nov13 ? 00:00:00 /opt/utran/moshell/monitor6054 Rnc99.utran.3g.singtel.com 33079 2. Kill the previous process if it's still running. kill -9 xxxxx xxxxx=process number related to particular Rnc [itk@sgp-ominf-itk Rnc04]$ kill -9 19182 [itk@sgp-ominf-itk Rnc04]$ ps -ef | grep monitor itk 14161 12198 0 17:55 pts/16 00:00:00 grep monitor itk 15217 1 0 Nov05 ? 00:07:06 /opt/utran/moshell/monitor6054 rnc01.utran.3g.singtel.com 33079 itk 16009 1 0 Nov05 ? 00:04:35 /opt/utran/moshell/monitor6054 rnc02.utran.3g.singtel.com 33079 itk 16314 1 0 Nov05 ? rnc03.utran.3g.singtel.com 33079 itk 16649 1 0 Nov05 ? rnc04.utran.3g.singtel.com 33079 itk 16955 1 0 Nov05 ? rnc05.utran.3g.singtel.com 33079 itk 17260 1 0 Nov05 ? rnc06.utran.3g.singtel.com 33079 00:09:57 /opt/utran/moshell/monitor6054 00:10:09 /opt/utran/moshell/monitor6054 00:08:55 /opt/utran/moshell/monitor6054 00:08:37 /opt/utran/moshell/monitor6054 3. Start the monitor again using startUehMonitors.sh Note: it's case sensisitve eg. startUehMonitor.sh Rnc04 [itk@sgp-ominf-itk Rnc04]$ startUehMonitors.sh Rnc99 Attempting to start monitors towards the following RNCs 'Rnc99' attaching to monitor on Rnc99: Successfully started logging Ueh Exceptions for RNC Rnc99 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (32) HealthCheck results are stored at following location in OSS-RC /var/opt/ericsson/nms_hc_server/resultfiles/ in UAS(citrix server) 10.252.75.19/ominf-uas1 /ossrc/ericsson/nms_hc_server/resultfiles in RANOS 10.252.75.11/ranos01 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (33) How to do continous ping from etmfx n it's output Take example rbs5144 and etmfx is in slot 3 Telnet to rbs Lhsh 000300 000300> EtHostMo_startPing -d 10.251.75.3 -h 1 -c 25 000300> (10.251.60.103) received echo reply from (10.251.75.3). icmpSeq=0 time=1 ms (10.251.60.103) received echo reply from (10.251.75.3). icmpSeq=1 time=1 ms (10.251.60.103) received echo reply from (10.251.75.3). icmpSeq=2 time=1 ms (10.251.60.103) received echo reply from (10.251.75.3). icmpSeq=3 time=1 ms (10.251.60.103) received echo reply from (10.251.75.3). icmpSeq=4 time=1 ms (10.251.60.103) received echo reply from (10.251.75.3). icmpSeq=5 time=1 ms (10.251.60.103) received echo reply from (10.251.75.3). icmpSeq=6 time=1 ms (10.251.60.103) received echo reply from (10.251.75.3). icmpSeq=7 time=1 ms (10.251.60.103) received echo reply from (10.251.75.3). icmpSeq=8 time=1 ms (10.251.60.103) received echo reply from (10.251.75.3). icmpSeq=9 time=1 ms (10.251.60.103) received echo reply from (10.251.75.3). icmpSeq=10 time=1 ms (10.251.60.103) received echo reply from (10.251.75.3). icmpSeq=11 time=1 ms (10.251.60.103) received echo reply from (10.251.75.3). icmpSeq=12 time=1 ms (10.251.60.103) received echo reply from (10.251.75.3). icmpSeq=13 time=1 ms (10.251.60.103) received echo reply from (10.251.75.3). icmpSeq=14 time=1 ms (10.251.60.103) received echo reply from (10.251.75.3). icmpSeq=15 time=1 ms (10.251.60.103) received echo reply from (10.251.75.3). icmpSeq=16 time=1 ms (10.251.60.103) received echo reply from (10.251.75.3). icmpSeq=17 time=1 ms (10.251.60.103) received echo reply from (10.251.75.3). icmpSeq=18 time=1 ms (10.251.60.103) received echo reply from (10.251.75.3). icmpSeq=19 time=1 ms (10.251.60.103) received echo reply from (10.251.75.3). icmpSeq=20 time=1 ms (10.251.60.103) received echo reply from (10.251.75.3). icmpSeq=21 time=1 ms (10.251.60.103) received echo reply from (10.251.75.3). icmpSeq=22 time=1 ms (10.251.60.103) received echo reply from (10.251.75.3). icmpSeq=23 time=1 ms (10.251.60.103) received echo reply from (10.251.75.3). icmpSeq=24 time=1 ms ----- 10.251.60.103 PING statistics ----25 packets transmitted from (10.251.60.103) to (10.251.75.3). Packets received/timed-out: 25/0 (Packet loss 0) round-trip (ms) min/avg/max: 1/1/1 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (34) To check distribution of ccSP, dcSP, pdrSP devices lh mod drh_ccrh_topdata lh mod drh_dcrh_topdata lh mod drh_pdrrh Central device handler lh ranapmp drhcendh cc lh ranapmp drhcendh dc lh ranapmp drhcendh pdr >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (35) The design documents can be found at \\esgtsnt021\common\2-3Gconvergence-design\2-3G Convergence Design (5 Yr Plan)\MS12\design\ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (36) RBS xml files path on OSS-RC. Not all RBS are invoked to gerenate XML files, only few are invoked. Need to check how that is done. / var/opt/ericsson/nms_umts_pms_seg/segment1/XML/SubNetwork=Rnc03/MeContext=3014_Shan grilLaAp >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (37) To check the FQDN to IP address mapping you will need to login to DNS server 10.252.75.13 user: root pass: xxxx then go to directory /var/named and look for file utran.3g.singtel.com You will find the FQDN to IP address mapping here. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (38) To see the disabled SPBs in a given RNC use following command lst spu=1 Disab or you can use command # lst spb ^0 or #lst spb >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (39) How to set parameters within a structure. To see the value of particular parameter within a structure you need to use @ symbol # hget UtranCell rateSelectionPsInteractive@ulPrefRate Similarly, to set or change one parameter in a structure you need to use "=" sign #lset UtranCell=5997 rateSelectionPsInteractive ulPrefRate=64 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (40) To check the node downtime \\esgtsnt021\Project\FSC-Field-Support\InstantAccess\Customer\SingTel Mobile\3G(Umts)\SingTel lgs s: parse system log lgv v: parse availability log lgsv 30 (/c/logfiles/systemlog/...) (/c/logfiles/cello/CELLO_AVAILABILITY_LOG.xml) # display the last 30 days of the system logs n availability log lgv 30 | grep NodeRestarted ( to check RNC restart start time) lgv 30 | grep -i " restart completed" ( to check the RNC restart completed time) lgv 30 | grep GPB ( to check GPB restarts) # this give module restart - unop and operational time dump list -a | sort +2 lgvs 30 | grep CRIT (to look for example "Node down") lgv 30 | grep ShutdownCommand (to check manualrestarts) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (41) How to tackle ISL alarms ? There was following alarm on RNC06 2010-01-29 18:34:09 m Switch Internal Link Port 0 Fault replaceable_unit_problem Subrack=ES-1,Slot=28,PlugInUnit=1,SwitchCoreUnit=1 2010-01-29 18:34:09 m Switch Internal Link Fault replaceable_unit_problem InternalLinkGroup=ES-1,SwitchInternalLink=ISL-1-B **** Note that SwithcInternalLink MO is only for ES subracks. **** =================================================================================== Proxy Adm State Op. State MO =================================================================================== 14601 1 (UNLOCKED) SwitchFabric=1,InternalLinkGroup=ES1,SwitchInternalLink=ISL-1-A # plane A side ISL 14602 1 (UNLOCKED) SwitchFabric=1,InternalLinkGroup=ES1,SwitchInternalLink=ISL-1-B # plane B side ISL 14603 1 (UNLOCKED) 1 (ENABLED) SwitchFabric=1,InternalLinkGroup=ISL-3 ## Corrosponds to ISL between MS and ES-3 14604 1 (UNLOCKED) SwitchFabric=1,InternalLinkGroup=ISL3,SwitchInternalLink=ISL-3-A 14605 1 (UNLOCKED) SwitchFabric=1,InternalLinkGroup=ISL3,SwitchInternalLink=ISL-3-B 14607 1 (UNLOCKED) SwitchFabric=1,InternalLinkGroup=ES2,SwitchInternalLink=ISL-2-A 14608 1 (UNLOCKED) SwitchFabric=1,InternalLinkGroup=ES2,SwitchInternalLink=ISL-2-B 14609 1 (UNLOCKED) 1 (ENABLED) SwitchFabric=1,InternalLinkGroup=ISL-4 ## Corrosponds to ISL between MS and ES-4 14610 1 (UNLOCKED) SwitchFabric=1,InternalLinkGroup=ISL4,SwitchInternalLink=ISL-4-B 14611 1 (UNLOCKED) SwitchFabric=1,InternalLinkGroup=ISL4,SwitchInternalLink=ISL-4-A 14614 1 (UNLOCKED) 1 (ENABLED) SwitchFabric=1,InternalLinkGroup=ISL-5 ## Corrosponds to ISL between MS and ES-3 14615 1 (UNLOCKED) SwitchFabric=1,InternalLinkGroup=ISL5,SwitchInternalLink=ISL-5-A 14616 1 (UNLOCKED) SwitchFabric=1,InternalLinkGroup=ISL5,SwitchInternalLink=ISL-5-B This alarm indicates that the ISL-1-B from MS to ES-1 is having problem. The port 0 corrosponds to port D on SCB. ***** To check the more information on the hardware errors use the following commands ****** $lhsh 000000 spashwinfo all # this gives a lot of useful information about clock, erros on Hw device etc $lhsh 000000 spashwinfo clk # give info about clock- HW Clock Status, Clock Settings and clock debug counters etc $lhsh 000000 spaspccinfo # this tells you about which one is active plane and there active ISL, whether its plane A or plane B ##As all ISL are connected to MS you need to check the spaspccinfo on corrosponding ES to check which ISL is active. Futhermore, if you need to check the state of ISL whether its carrying traffic or not use this command $get ISL statestraffic =================================================================================== ========== InternalLinkGroup=ES-2,SwitchInternalLink=ISL-2-A operationalStateTraffic 1 (ENABLED) InternalLinkGroup=ES-2,SwitchInternalLink=ISL-2-B operationalStateTraffic 1 (ENABLED) InternalLinkGroup=ES-1,SwitchInternalLink=ISL-1-A operationalStateTraffic 1 (ENABLED) InternalLinkGroup=ES-1,SwitchInternalLink=ISL-1-B operationalStateTraffic 1 (ENABLED) InternalLinkGroup=ISL-3,SwitchInternalLink=ISL-3-A operationalStateTraffic 1 (ENABLED) InternalLinkGroup=ISL-3,SwitchInternalLink=ISL-3-B operationalStateTraffic 1 (ENABLED) =================================================================================== ========== ISL load counters are present in RANAP/SCCP board. In 3810 its on 001700, in 3820 its on 000700. The ISL load alarm is monitored by "drhcendh" program running on RANAP board. You will see ISL load alarm in RANAP board not in SCB board. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> (42) About SPB boards - How are repertoire, SwAllocation, Slot, SPU, SPM and cc/dc/pdr devices are related -----------------repertoire -->SwAllocation=SPB_TYPE_A-->Equipment, Slot=MS-9 -->Equipment=1,Subrack=MS,Slot=9,PlugInUnit=1,Spu=1,Spm=2---> this refers to the corrosponding DC, CC or PDR device 1) Repertoire is defined only once Ex RNC_SPB_TYPE_A 2) Repertoire is reserved by certain "SwAllocation" such as SPB_TYPE_A 3) "SwAllocation" is reserved by certain slots. eg. slot=MS-9 4) Each SPB slot has SPU e.g. Equipment=1,Subrack=MS,Slot=9,PlugInUnit=1,Spu=1 5) Each Spu has SPM ( special processor module) e.g.Equipment=1,Subrack=MS,Slot=9,PlugInUnit=1,Spu=1, Spm=5 6) Now this SPM=5 refers to a DCdevice . What is the significance of Control Plane status in SCB board and how does it work? Sctp=ES2-2 Location of documents in Eridoc Cabinets/WorkGroups/ENO Services/PMO/FCP 121 1977 - SingTel Mobile Projects ENO11/Current Projects/FCP 121 1869 - 2G_3G Core Network Expansion FY 09_10 Implementation Projects/2_FCP 121 1869 - RAN Network Expansion Projects/2_FCP 121 1869_5 - RNC7 & 8 New Nodes (3820)/3. Project Execution Phase/3.4 Technical Documents/RNC7 & 8 (3820) UAT Enabling traces for Ans_aal2ap_proc to identify which ETMF4 is carrying CS traffic at the given moment. [ericsson@ompc02 ~]$ /opt/utran/moshell/monitor6054 10.251.88.228 33080 > RNC98_aal2path_20100423.log & [ericsson@ompc02 ~]$ tail -f RNC98_aal2path_20100423.log | concat_segmented_rlib_traces.pl | decoder.pl --w10a | flow.pl -color >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> (43) IPAC_PING In rbs there is no more ipaccesshostgpb. Its internally communicated using ipaccesshostet. So to check packet losses from IPAccessHostGPB to CP/UP of RBS do the following # Get the ip address of any ipaccesshostgpb in rnc. This will be source IP address - 10.251.66.23 # get IpaccesshostEt of RBS this would be destination address ex - 10.251.61.30 Use command Control plane pinging ---------------------- ipac_ping -z 10.251.66.23 -d 10.251.61.30 -c 20 //20 packets ping from rnc to rbs User plane pinging -------------------Get ipaddress of ipaccesshostet in rnc #lacc IpSystem=1,IpAccessHostEt=ES-3-4-1 ping // to the CP/UP of the RBS. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> (44) THP traces 1. Enable normal traces in RNC corresponding module lh mod15 te default lh mod15 te e bus_send bus_receive NBAP_ASN lh mod15 te e bus_send bus_receive RRC_ASN lh mod15 te e bus_send bus_receive RANAP_ASN lh mod15 te e bus_send bus_receive Ans* lh mod15 te filter set "[1]<>8" NBAP* 2. Trace for checking the mapped SPI lh mod15 te e all UE_THP_MAP 3. Then activate monitor on RNC module ..monitor33057 10.251.88.51 | ./decoder.pl | tee log1.txt | ./flow.pl -colour | tee log2.txt 4. In RAB Assignment Accept IE or PDP context Accept IE, we can see the THP/ARP received from Core Network. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> (45) the Alarm Log does not work in EM for RBS. 3402_CargoAgent_BldgE> pwd 100601-13:58:30 10.102.7.241 8.0g RBS_NODE_MODEL_L_10_9 stopfile=/tmp/25052 $ pwd /c/logfiles/alarm_event/ <==== this is location of ALARM_LOG.xml file. $ 3402_CargoAgent_BldgE> ls 100601-13:58:37 10.102.7.241 8.0g RBS_NODE_MODEL_L_10_9 stopfile=/tmp/25052 $ ls Directory '/c/logfiles/alarm_event/' ALARM_LOG.xml EVENT_LOG.xml Remedy Rename the file ALARM_LOG.xml to ALARM_LOG_corrupted.xml. Then cold restart the RBS. New ALARM_LOG.xml file will be generated and EM should be able to lauch Alarm_Log. SCS921271 1. Login to RBS using amos 2. go to directory using commad $cd /c/logfiles/alarm_event/ 3. In this directory run following commad $mv /c/logfiles/alarm_event/ALARM_LOG.xml /c/logfiles/alarm_event/ALARM_LOG_corrupted.xml 4. Cold restart rbs. 5. After restart check EMAS, you should be able to launch alarm log through EMAS 6. Again go to the directory /c/logfiles/alarm_event/ and run following command $ rm /c/logfiles/alarm_event/ALARM_LOG_corrupted.xml >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> (47) Checking corruption in database [ericsson@ompc02 evisjag]$ moshell -d db.dat #################################################################### # Welcome to MoShell 8.0g # # Finn Magnusson, Jan Pettersson # # http://utran01.au.ao.ericsson.se/moshell # # Contact: [email protected] # # [email protected] # #################################################################### Starting Polyhedra SQL server on port 8001... OK . Reading PM scanner data (2 lines).. Reading MAO data (1767 lines) ... Connected to OFFLINE_db.dat, Node: 3462_JurongEastBlk341 (SubNetwork=STM_R,SubNetwork=Rnc06,MeContext=3462_JurongEastBlk341,ManagedElement=1 ) MOM version: RBS_NODE_MODEL_L_10_9 (CPP7) 0% ~50% ~100% ................................................................................... ................. Last MO: 1766. Loaded 1766 MOs. Total: 1767 MOs. Reading FRO data (459 lines) ... 0% ~50% ~100% ................................................................................... ................ The database is OK To print the consistency check result, type "dbc". To print the list of available commands, type "h". HELP MENU QUIT : h : q OFFLINE_DB.DAT> dbc 100602-19:05:24 OFFLINE_db.dat 8.0g RBS_NODE_MODEL_L_10_9 stopfile=/tmp/28586 ============================================================================== Database Consistency Check for node: 3462_JurongEastBlk341, ip=10.106.2.113, mom=RBS_L_10_9, up=CXP9012959/1_R9CT01 P7 File: db.dat ============================================================================== 1) MAOs with unauthorized characters in the MAO name (allowed range: -_/.A-Za-z09!%:*): NO -----------------------------------------------------------------------------2) MAOs with dangerous characters in the MAO name `,=�^"|'� (HL11572/UABtr75948): NO -----------------------------------------------------------------------------3) MAOs without FROs: NO -----------------------------------------------------------------------------4) FROs without MAOs: NO -----------------------------------------------------------------------------5) MAOs with duplicate LDN: NO -----------------------------------------------------------------------------6) MAOs with duplicate froId: NO -----------------------------------------------------------------------------7) Mismatch between number of MAO and FRO instances: NO -----------------------------------------------------------------------------8) MAOs referring to non-existent MAOs: NO -----------------------------------------------------------------------------9) MAOs defined under a different parent than FRO: NO -----------------------------------------------------------------------------10) Inconsistent MO references between MAO and FRO: NO -----------------------------------------------------------------------------11) MAOs missing from reservedBy list: NO -----------------------------------------------------------------------------12) MAOs hanging in reservedBy list: NO -----------------------------------------------------------------------------13) Inconsistent sequence of moRefs in MAO: NO -----------------------------------------------------------------------------Conclusion: the database is OK >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> (48) Terminal Server Access to RNC Login to term server - ericsson/eric123 then telent to rnc using rnc/rnc 10.252.147.3 7001 RNC1 slot 10 10.252.147.3 7002 RNC1 slot 11 10.252.147.3 7003 RNC2 slot 10 10.252.147.3 7004 RNC2 slot 11 10.252.147.3 7005 RNC3 slot 10 10.252.147.3 7006 RNC3 slot 11 10.252.75.70 7001 RNC4 slot 10 10.252.75.70 7002 RNC4 slot 11 10.252.75.70 7003 RNC5 slot 10 10.252.75.70 7004 RNC5 slot 11 10.252.75.70 7005 RNC6 slot 10 10.252.75.70 7006 RNC6 slot 11 10.252.75.70 7012 RNC99 slot 10 10.252.75.70 7013 RNC99 slot 11 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> (49) How to check whether RRU is still loading software or not ? $ lhsh 001200/port_0_dev_7 lmclist 100608-00:30:35 10.251.132.138 8.0g RBS_NODE_MODEL_M_1_8 $ lhsh 001200/port_0_dev_7 lmclist State Size Curr ProdDate Seq Type LMs 0 Working 1572864 Yes 2007-04-26 15:00 0 Boot 4 1 Working 7340032 No 2009-06-12 14:00 0 Applic 7 2 Writing <========its writting software $ stopfile=/tmp/16548 SwPid CXP901096%7_R1D01 CXP9012183%6_R2G01 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> (50) How to check where ftc is active and which board is active. Rnc02> cab 100608-04:57:47 10.251.131.35 8.0g RNC_NODE_MODEL_L_1_57 stopfile=/tmp/13125 put /home/ericsson/moshell_logfiles/logs_moshell/tempfiles/20100607224650_13096/lhCmd13125 /d/usr/lhCmd13125 ... OK .................................................put /home/ericsson/moshell_logfiles/logs_moshell/tempfiles/20100607224650_13096/lhCmd13125 /d/usr/lhCmd13125 ... OK .......................................................................... =================================================================================== ================== SMN APN BOARD GREEN YELLOW RED PRODUCT NR REV DATE SERIAL NR TEMP COREMGR =================================================================================== ================== 0 1 SCB3 ON OFF OFF ROJ1192108/4 R2C/B 20080312 TU89835666 +25C 0 2 SXB3 ON OFF OFF ROJ1192109/3 R1C 20060724 TU89292751 +23C 0 3 SXB3 ON OFF OFF ROJ1192109/3 R1C 20060724 TU89292752 +22C 0 4 TUB2 ON OFF OFF ROJ1192104/4 R3C 20080401 TU8G183859 0 5 TUB2 ON OFF OFF ROJ1192104/4 R3C 20080401 TU8G183739 0 6 ET-M4 ON OFF OFF ROJ1192102/1 R3G 03W24 S952162858 0 7 ET-M4 S952162872 0 8 GPB53 A400705648 +28C 0 9 GPB53 A400705813 +28C 0 10 GPB53 TU8BW25374 +28C 0 11 GPB53 TU88791523 +28C 0 12 GPB53 TU88758393 +27C 0 13 GPB53 TU88791558 +28C ON OFF OFF ROJ1192102/1 R3G 03W24 ON OFF OFF ROJ1192106/53 R4D 20080408 ON OFF OFF ROJ1192106/53 R4D 20080408 ON OFF Active* ON OFF StandbyReady ON OFF OFF ROJ1192106/53 R4D 20070824 OFF ROJ1192106/53 R4B 20060519 OFF ROJ1192106/53 R4B 20060516 OFF ROJ1192106/53 R4B 20060519 ON OFF * represents which processes are active Active/StandbyReady - FTC active where >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> (51) How to check admission fault using admission history log ? COLI command to extract the admission history log is: lhsh <modMP> admhistory -c <cId> printlog This is a circular log containiing the last 128 admission events in the cell. Once an admision fault is detected, the log is frozen. One can see the state of the log in the header of the printout. It is says "Not frozen", no fault has been detected. If it says "Frozen at ...", an admission fault has been detected and the log will contain information about the fault. Once the log has been extracted, it can be reset using command lhsh <modMP> admhistory -c <cId> reset The log will then be unfrozen to detect the next admission fault occuring. lh lh lh lh lh lh mod182 mod182 mod182 mod182 mod182 mod182 admhistory admhistory admhistory admhistory admhistory admhistory -c -c -c -c -c -c 31031 31032 31033 31034 31035 31036 printlog printlog printlog printlog printlog printlog lh lh lh lh lh lh mod192 mod192 mod192 mod192 mod192 mod192 admhistory admhistory admhistory admhistory admhistory admhistory -c -c -c -c -c -c 40151 40152 40153 40154 40155 40156 printlog printlog printlog printlog printlog printlog lh mod110 admhistory -c 30912 printlog lh mod110 admhistory -c 30915 printlog <------------------------------------------RBS TRACES Start------------------------------------------> Enable traces in RBS on RAX board ================================== lh rax te e trace3 TC_MISC lh rax te e all DEM0_BP lh rax te e all DEM1_BP lh rax te e all DEM2_BP lh rax te e all DEM3_BP lh lh lh lh rax rax rax rax te te te te e e e e trace1 trace1 trace1 trace1 trace2 trace2 trace2 trace2 DEM0_WS DEM1_WS DEM2_WS DEM3_WS lh lh lh lh rax rax rax rax te te te te e e e e trace1 trace1 trace1 trace1 trace2 trace2 trace2 trace2 DEM0_RAKE DEM1_RAKE DEM2_RAKE DEM3_RAKE lh lh lh lh rax rax rax rax te te te te e e e e trace1 trace1 trace1 trace1 DEM0_MDS DEM1_MDS DEM2_MDS DEM3_MDS lh lh lh lh rax rax rax rax te te te te e e e e trace2 trace2 trace2 trace2 DEM0_CR DEM1_CR DEM2_CR DEM3_CR lh lh lh lh rax rax rax rax te te te te e e e e trace2 trace2 trace2 trace2 DEM0_DM DEM1_DM DEM2_DM DEM3_DM Enable traces in RBS on TX board ================================== lh tx te e trace5 DCHSR1_FACH lh tx te e trace5 DCHSR2_FACH lh tx te e trace5 DCHSR3_FACH lh tx te e trace5 DCHSR4_FACH lh tx te e trace5 DCHSR5_FACH lh lh lh lh lh tx tx tx tx tx te te te te te e e e e e trace6 trace6 trace6 trace6 trace6 DCHSR1_dfop DCHSR2_dfop DCHSR3_dfop DCHSR4_dfop DCHSR5_dfop lh lh lh lh lh tx tx tx tx tx te te te te te e e e e e trace2 trace2 trace2 trace2 trace2 DCHSR1_dfop DCHSR2_dfop DCHSR3_dfop DCHSR4_dfop DCHSR5_dfop lh lh lh lh tx tx tx tx te te te te e e e e trace3 trace3 trace3 trace3 DCHSR1_CRI_sslrq DCHSR2_CRI_sslrq DCHSR3_CRI_sslrq DCHSR4_CRI_sslrq lh tx te e trace3 DCHSR5_CRI_sslrq Disable traces ================================== lh rax te default lh tx te default Collect traces using following commands by Andres/ Johnny ============================================================ lh rax te log read lh tx te log read <------------------------------------------RBS TRACES end--------------------------------------------> <------------------------------------------RNC TRACES Start------------------------------------------> enable traces ============= ## Cell take on module MP lh mod182 cellenable -cid 31031 # for problemetic cells lh mod182 cellenable -cid 31032 # for problemetic cells lh mod182 cellenable -cid 31033 # for problemetic cells lh lh lh lh mod182 mod182 mod182 mod182 uerandtrace uerandtrace uerandtrace uerandtrace lh lh lh lh lh mod182 mod182 mod182 mod182 mod182 te te te te te e e e e e on -cell cid 31031 on -cell cid 31032 on -cell cid 31033 max 6 trace7 ADMISSION_CELL all ADMISSION_REJECT_CELL trace4 UE_GENERAL trace1 UEH_EXCEPTION trace9 ADMISSION disable traces =============== lh mod182 te default lh mod182 te e all UEH_EXCEPTION lh mod182 te e trace1 RNC_EXCEPTION lh mod182 celldisable -cid 31034 lh mod182 celldisable -cid 31035 lh mod182 celldisable -cid 31036 lh mod182 uerandtrace off lh mod182 ueidtrace off Log collection which will be done by Anders ================================= lh mod182 hwm -c 31031 print rbs lh mod182 hwm -c 31032 print rbs lh mod182 hwm -c 31033 print rbs lh lh lh lh mod182 mod182 mod182 mod182 celldata celldata celldata celldata -c -c -c -c 31031 31032 31033 31034 lh mod182 celldata -c 31035 lh mod182 celldata -c 31036 lh mod182 hwm -c 31031 print grp lh mod182 hwm -c 31032 print grp lh mod182 hwm -c 31033 print grp lh lh lh lh lh lh mod182 mod182 mod182 mod182 mod182 mod182 codetree codetree codetree codetree codetree codetree -cid -cid -cid -cid -cid -cid 31031 31032 31033 31034 31035 31036 lh mod182 te log read lh mod182 admhistory -c 31031 printlog lh mod182 admhistory -c 31032 printlog lh mod182 admhistory -c 31033 printlog <------------------------------------------RNC TRACES end------------------------------------------> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> (52) How to check the RBS CE utilization instead of setting dlHwAdm to 99 or ulHwAdm to 99 ? The counters to use for CE monitoring in RBS are: MO: DownlinkBaseBandPool Counter: pmCapacityDlCe MO: UplinkBaseBandPool Counter: pmCapacityUlCe There are also other counters in the same MO's that can be used in different ways to monitor capacity usage. Counters for monitoring UL/DL HW in RBS can be found in CPI for RBS in performance measurment: http://cpi.al.sw.ericsson.se/alexserv?ac=LINKEXT&li=EN/LZN7350314R1B&FN=5_1551LZA7016003Uen.D.html >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> (53) Summary of dc-sp resource allocation algorithm RncFunction::evolvedHsUePrioEnabled ----------------------------------Controls if prioritization of Evolved HS UEs is taken into account when allocating UEs to DC-SPs in the RNC node. When the parameter is set to TRUE, the priority of UEs with different HS capabilities is set with the parameter evolvedHsUePrioLevel. When the parameter is set to FALSE, all HS capable UEs are allocated to the SP with most available resources in the module, and Non-HS capable UEs are allocated to the SP with second most resources. RncFunction:evolvedHsUePrioLevel ---------------------------------- Controls the priority of different HS capable UEs at DC-SP allocation in the RNC node. Each element in the sequence represents a priority level used for allocating UEs to DC-SPs. The value of each element specifies the minimum UE capability that is included in that priority. Priority level 1 UEs are allocated to the SP with the most available resources, priority level 2 UEs are allocated to the SP with the second most available resources, and so on. The capability used for priority level 1 must be a higher capability than the capability for priority level 2, and so on. The ordering of UE capabilities is defined by the enumeration SupportedHsUeCategories. Higher values in this enumeration indicate a higher capability. UEs below priority level 2 are allocated to the SP that is least loaded among the remaining ones. Dependencies: This parameter is not used when evolvedHsUePrioEnabled is set to FALSE. When new UE request DC resources, it is calculated which DC SP will be assigned according to configuration and following table: Ue capability evolvedHsUePrioEnabled evolvedHsUePrioLevel[ ] SP allocation prioUeGroup ---------------------------------------------------------------------------------------------------------------------------NON_HS_CAPABLE OFF *(the parameter is not used) Second least loaded SP HS HS_CAPABLE OFF *(the parameter is not used) Least loaded SP HS EVOLVED_HS_CAPABLE OFF *(the parameter is not used) Least loaded SP HS MULTI_CARRIER_CAPABLE OFF *(the parameter is not used) Least loaded SP HS NON_HS_CAPABLE ON [1]:BASIC_HS[2]:NON_HS Second least loaded SP HS HS_CAPABLE ON [1]:BASIC_HS[2]:NON_HS Least loaded SP HS EVOLVED_HS_CAPABLE ON [1]:BASIC_HS[2]:NON_HS Least loaded SP HS MULTI_CARRIER_CAPABLE ON [1]:BASIC_HS[2]:NON_HS Least loaded SP HS NON_HS_CAPABLE ON [1]:MIMO_64QAM[2]:NON_HS Second least loaded SP HS_EVOLVED HS_CAPABLE ON [1]:MIMO_64QAM[2]:NON_HS Second least loaded SP HS_EVOLVED EVOLVED_HS_CAPABLE ON [1]:MIMO_64QAM[2]:NON_HS Least loaded SP HS_EVOLVED MULTI_CARRIER_CAPABLE ON [1]:MIMO_64QAM[2]:NON_HS Least loaded SP HS_EVOLVED NON_HS_CAPABLE ON [1]:MIMO_64QAM[2]:BASIC_HS Third least loaded SP HS_EVOLVED HS_CAPABLE ON [1]:MIMO_64QAM[2]:BASIC_HS Second least loaded SP HS_EVOLVED EVOLVED_HS_CAPABLE ON [1]:MIMO_64QAM[2]:BASIC_HS Least loaded SP HS_EVOLVED MULTI_CARRIER_CAPABLE ON [1]:MIMO_64QAM[2]:BASIC_HS Least loaded SP HS_EVOLVED NON_HS_CAPABLE ON [1]:MULTI_CARRIER[2]:MIMO_64QAM Third least loaded SP MULTI_CARRIER HS_CAPABLE ON [1]:MULTI_CARRIER[2]:MIMO_64QAM Third least loaded SP MULTI_CARRIER EVOLVED_HS_CAPABLE ON [1]:MULTI_CARRIER[2]:MIMO_64QAM Second least loaded SP MULTI_CARRIER MULTI_CARRIER_CAPABLE ON [1]:MULTI_CARRIER[2]:MIMO_64QAM Least loaded SP MULTI_CARRIER >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> (54) How to restore CV of a live node into OLP/STP Step 1 Get Backup the CV from the live Node Rnc> cvget �R9SB_P710_137_20100525� A zipped file R9SB_P710_137_20100525.zip will be created and stored on local machine. e.g. Rnc99> cvget R9SB_P710_23_20100323 100617-13:17:14 10.251.88.52 8.0g RNC_NODE_MODEL_L_1_57 stopfile=/tmp/29591 Download the following 1 CV's to /home/ericsson/moshell_logfiles/logs_moshell/cv/RNC99/100617_131714? =================================================================================== ======================= R9SB_P710_23_20100323 2010-03-23 06:40 CXP9013831_R9SB earmcat External Cell 59981 =================================================================================== ======================= Are you Sure [y/n] ? y Downloading CV 1 of 1: R9SB_P710_23_20100323 facc configurationversion=1 putToFtpServer R9SB_P710_23_20100323 /d/usr RNC99_R9SB_P710_23_20100323.zip 127.0.0.1 moshelluser $password =================================================================================== ============================== Proxy MO Action Nr of Params =================================================================================== ============================== 3520 ConfigurationVersion=1 putToFtpServer 6 >>> Return value = 51723335 =================================================================================== ============================== Total: 1 MOs attempted, 1 MOs actioned currentMainActivity EXPORTING_BACKUP_CV IDLE Waiting 20 seconds before starting to poll... ConfigurationVersion=1 ElapsedTime currentDetailedActivity ConfigurationVersion=1 0s CREATING_BACKUP ConfigurationVersion=1 10s IDLE Waiting 60 seconds before checking result... ConfigurationVersion=1 70s (01m10s) IDLE IDLE CV action result: EXECUTED additionalInfo : CV_BACKUP_NAME RNC99_R9SB_P710_23_20100323.zip Get /d/usr/RNC99_R9SB_P710_23_20100323.zip /home/ericsson/moshell_logfiles/logs_moshell/cv/RNC99/100617_131714/RNC99_R9SB_P710 _23_20100323.zip ... OK Dele /d/usr/RNC99_R9SB_P710_23_20100323.zip ... OK Successful storage of R9SB_P710_23_20100323 to /home/ericsson/moshell_logfiles/logs_moshell/cv/RNC99/100617_131714/RNC99_R9SB_P710 _23_20100323.zip Step 2 Make sure OLP RNC is not connected to any live node so that restoring CV will not cause the traffic disturbance Pay attention to all RNC interfaces including RANOS especially at ETMFG-24, 25, ETM-4 6, 7, ES1-4 & 25 (RANOS). Step 3 RNC> cvput R9SB_P710_137_20100525.zip Once completed, the CV will appear in �cvls� and state download Step 4 Restore the CV using following action RNC> acc configuration=1 forcedrestore Use forcedrestore when restoring a different node CV, select autoconfigurationon to 1 or ON The node will restart on the restored CV, the node ip address will be changed to that of the node in STP, no further action is required. Step 5 Confirm restore once forcedrestore is completed RNC> acc configuration=1 confirmrestore This step must be done or else it can prevent other cv command to fail >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> (55) UETR traces are stored in following directory in OSS-RC /var/opt/ericsson/nms_umts_pms_seg/segment1/UETR >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> (56) RANAP/RNSAP Separation commands on RNC3820 On RNC3820 RNSAP active/passive board change ----------------------------------------------------------------------get rnc_rnsap_c2 set ReliableProgramUniter=rnc_rnsap_c2 admActiveSlot Subrack=MS,Slot=22 set ReliableProgramUniter=rnc_rnsap_c2 admPassiveSlot Subrack=MS,Slot=7 ----------------------------------------------------------------------Normalization set ReliableProgramUniter=rnc_ranap_c2 normalisation 2 set ReliableProgramUniter=rnc_rnsap_c2 normalisation 2 set ReliableProgramUniter=rnc_cen_drh_c2 normalisation 2 set ReliableProgramUniter=rnc_cen_agps_c2 normalisation 2 set ReliableProgramUniter=rnc_cen_mbms_c2 normalisation 2 set ReliableProgramUniter=rnc_cen_rnh_c2 normalisation 2 set ReliableProgramUniter=rnc_pcap_c2 normalisation 2 set ReliableProgramUniter=rnc_sabp_c2 normalisation 2 -----------------------------------------------------------------------Replication lset SwManagement=1,ReliableProgramUniter=rnc_ranap_c2 replication 3 lset SwManagement=1,ReliableProgramUniter=rnc_rnsap_c2 replication 3 --------------------------------------------------------------------->>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> (57) UEH Handling [2010-10-15 03:30:30.380] 001400/RncLmUePT(UEH_EXCEPTION) ../src/UehChSwitchFachToDchC.cpp:5597 TRACE1:ExceptionCode = 310; Received signal NBAP RADIO LINK SETUP FAILURE ; FailureIndication; RBS; RRCConRel; Dropped Call Release Pending; UehChSwitchFachToDchC; UeRef = 4113; IMSI = 525016180187902; cellId = 30384; cellFroId = 398; S-RNTI = 20497; connType=uehPacketCch; sGCP=00000000000; reqConnType=uehPacketEulHs; tGCP=01000000000; causeCode=4; rlSetupFail received (Top::ProcedureCoordination::FachToDchSwitchingSrnc::NbapRlSetupProcedure::nbapRlSe tup) Explanation Exception happed in subsystem "RncLmUePT". ExceptionCode = 310; Perl script for the UEH_EXCEPTION handling UEsift.pl [-h|-r|-d|-f] [-ue XXX] UEsift is aimed at sifting on massages for specific UEs. UEsift is aimed to work with UEH_EXCEPTIONs and the main protocol ASN bus_send bus_receive traces on modules, ie 'ueRef = XXX' or 'UeRef = XXX' . Updated to also work with DC traces, ie 'ueref XXX' Can be used at different levels, raw log & decoded or flowed logs for modules See UEsift.pl -h for details Examples: cat file.log | concat_segmented_rlib_traces.pl | UEsift.pl -r -ue XXXX cat file.log | concat_segmented_rlib_traces.pl | decoder.pl --pX | UEsift.pl -d -ue XXXX cat file.log | concat_segmented_rlib_traces.pl | p6decoder --PX | flow.pl -colour | ~/scripts/UEsift.pl -f -ue XXXX >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> (58) Loading UP package into 10.252.75.14 OSS-RC Master server. FTP the UP to 10.252.75.14. smrsuser/singtel unzip package in oss-rc master server Synchronize the CPP SWP in SMO Import the package in SMO if required. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> (60) Restarting a program i.e. LM is better than restarting a process. e.g. #lh mod60 pm_pginfo -a or #lh mod60 sma -all or #lh mod60 listloaded 0006: Install handle: CXC1727645%1_R4S01 (RncLmUe) 0006: Entry point: 468ef414 0006: Base address of the text region: 45c20000 0006: Size of the text region (bytes): 025e5000 0006: Base address of the data region: 48210000 0006: Size of the data region (bytes): 01241000 0006: Number of sections in the load module: 21 0006: Load module id: CXC1727619%1_R4S01 (RncLmCell) 0006: Load module path: @(#)/vobs/rnc/rrt/rncNode/rncSys/rncLm/rncLmCellLmU/model/RncLmCellLmU_mp750_EXE/bu ild RncLmCell ppc elmcm002_lmbuild_wanessamd_cell_standard_1.0 0006: Load module version: @(#)CXC1727619/1 R4S01 2010-09-06 08:02:15 GMT (RncLmCell) 0006: Install handle: CXC1727619%1_R4S01 (RncLmCell) 0006: Entry point: 4b4a45ec 0006: Base address of the text region: 4ac10000 0006: Size of the text region (bytes): 01698000 0006: Base address of the data region: 4c2b0000 0006: Size of the data region (bytes): 006b0000 0006: Number of sections in the load module: 21 0006: Load module id: CXC1727623%1_R4S01 (RncLmLoadCtr) 0006: Load module path: @(#)/vobs/rnc/rrt/rncNode/rncSys/rncLm/rncLmLoadCtrLmU/model/RncLmLoadCtrLmU_mp750_ EXE/build RncLmLoadCtr ppc elmcm002_lmbuild_wanessamd_loadctr_standard_1.0 # To search the program corrosponding to "CXC1727619%1_R4S01" replace % with / "CXC1727619/1_R4S01" # lpr CXC1727619/1_R4S01 Rnc98> lpr CXC1727619/1_R4S01 101012-15:14:35 10.251.88.228 8.0g RNC_NODE_MODEL_L_1_62 stopfile=/tmp/22507 =================================================================================== Proxy MO =================================================================================== 488 Equipment=1,Subrack=ES1,Slot=2,PlugInUnit=1,Program=CXC1727619/1_R4S01 751 Equipment=1,Subrack=MS,Slot=8,PlugInUnit=1,Program=CXC1727619/1_R4S01 1191 Equipment=1,Subrack=MS,Slot=6,PlugInUnit=1,Program=CXC1727619/1_R4S01 1378 SwManagement=1,LoadModule=CXC1727619/1_R4S01 =================================================================================== Total: 4 MOs Rnc98> acl 488 101012-15:14:44 10.251.88.228 8.0g RNC_NODE_MODEL_L_1_62 stopfile=/tmp/22507 =================================================================================== ============================== Proxy MO Action Nr of Params =================================================================================== ============================== 488 Subrack=ES1,Slot=2,PlugInUnit=1,Program=CXC1727619/1_R4S01 restart 0 =========================================================================== #acc 488 restart # lh mod60 pm_pginfo -a 0006: Install handle : CXC1727619%1_R4S01 0006: Program pid : 0x000201f2 0006: Domain : 0x00000013 0006: Segment pid : 0x00010284 0006: Main block : 0x000201f2 0006: Main process : 0x00010286 (RncLmCell) 0006: 0006: 0006: 0006: 0006: 0006: 0006: 0006: 0006: 0006: 0006: 0006: 0006: 0006: 0006: 0006: 0006: 0006: 0006: 0006: 0006: 0006: 0006: Stk poolid : 0x00010285 Stk pool base : 0x06acf000 Stk pool size : 0x0097dd20 Sig poolid : 0x00010285 Sig pool base : 0x06acf000 Sig pool size : 0x0097dd20 UID : 0x00000000 Heap ref : 0x009763a1 State : started Install handle : CXC1727645%1_R4S01 0006: Program pid : 0x00030174 Domain : 0x00000012 Segment pid : 0x000401a0 Main block : 0x00030174 Main process : 0x000101a4 Stk poolid : 0x000101a3 Stk pool base : 0x05e48000 Stk pool size : 0x00c861c0 Sig poolid : 0x000101a3 Sig pool base : 0x05e48000 Sig pool size : 0x00c861c0 UID : 0x00000000 Heap ref : 0x009659e1 State : started (RncLmUe) To kill the program use the following command # lh mod60 pm_kill "0x000201f2" To check the processes running in a modules #lh modxx ps >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> (61) to get the cc, dc, pdr allocation to the module use the following commands # bp mod # bp cc # bp dc # bp pdr >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> (62) To see the cc, dc, pdr distribution status you can use following commands # std The best way is to see the central device handler located in RANAP board # lh ranapmp drhcendh cc # lh ranapmp drhcendh dc # lh ranapmp drhcendh pdr >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> (63) Analyzing pmd file >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> (64) Upgrade failure analysis When you install package it is installed in /d drive Use Full package installation generally Use "lgu" to check the log. To see the exception logs in lgu filter the log using "UP-E" The installed LM reside in /d/loadmodules The upgraded LM resides in /c/up/ Rnc98> cd /c/up/ 101013-11:15:29 10.251.88.228 8.0g RNC_NODE_MODEL_L_1_62 stopfile=/tmp/14442 $ cd /c/up/ $ Rnc98> ls -ltr 101013-11:15:32 10.251.88.228 8.0g RNC_NODE_MODEL_L_1_62 stopfile=/tmp/14442 $ ls -ltr Directory '/c/up/' drwxrwxrwx 0 512 5 Aug 2010 19:19:49 CXP9014711%1_R4F drwxrwxrwx 0 512 17 Jun 2010 20:08:53 CXP9014711%1_R3A (CPP EP13-7.0.3.13) drwxrwxrwx 0 512 12 Apr 2010 04:32:16 CXP9014711%1_R2C (EP9-1-CPP_7.0.3.13) drwxrwxrwx 0 512 19 Feb 2010 12:32:56 CXP9014711%1_R1V (CPP_7.0.3.7-EP12) drwxrwxrwx 0 512 27 Aug 2010 04:19:31 CXP9014711%1_R3E drwxrwxrwx 0 512 10 Oct 2010 23:22:48 CXP9014711%1_R4S $ W10 W10.0.2 W10.0.1-1 W10.0 W10 W10 The UCF (upgrade control file) resides in corresponding UP. Rnc98> cd CXP9014711%1_R4S 101013-11:16:19 10.251.88.228 8.0g RNC_NODE_MODEL_L_1_62 stopfile=/tmp/14442 $ cd CXP9014711%1_R4S $ Rnc98> ls -ltr 101013-11:16:24 10.251.88.228 8.0g RNC_NODE_MODEL_L_1_62 stopfile=/tmp/14442 $ ls -ltr Directory '/c/up/CXP9014711%1_R4S/' W10 -rw-rw-rw0 431653 10 Oct 2010 23:22:48 CXP9014711%1_R4S.xml W10 -rw-rw-rw0 431653 10 Oct 2010 23:03:04 CXP9014711%1_R4S.xmlOriginal W10 $ UCF is used to upgrade each board "from" state to "to" state. It contains all the information about the compatible HW and possible LM that support this type of hardware. Hence it is mandatory to check the release notes to see if the RNC contains mandatory HW required by the UP. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> (65) Generally following traces would be enabled. Position Board AddedTraceConditions 0102 GPB65 0102 GPB65 Process Status RNC_EXCEPTION UE_ASN_RRC enabled enabled trace1 bus_send bus_receive 0102 GPB65 0102 GPB65 bus_send bus_receive 0102 GPB65 bus_send bus_receive 0102 GPB65 bus_send bus_receive 0102 GPB65 0102 GPB65 trace3 ==> gives info 0102 GPB65 0102 GPB65 UEH_EXCEPTION UE_ASN_NBAP enabled enabled trace1 UE_ASN_RANAP enabled UE_ASN_RNSAP enabled UE_GENERAL UE_IU_IF about the cause for system release uehUeCtxtC[77] uehDrncDedChRlCommonC enabled enabled trace4 trace1 enabled enabled user1 user1 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> (66) To check the sctp host unrechable at rbs side. 3719_YishunBlk406> facc ipaccesshostet ping 10.251.75.36 -List the SCTP unreachable association 3719_YishunBlk406> sctphost_info -assoc -unreach -list >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> (67) $ lh mod60 ue_poolinfo >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> (68) Using rrt commands $tgrc => It gives you actor list for each cell. Print Resource Object information for all MOs in LmCell (RNC only). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> (69) UEH_EXCEPTION analysis - Usually used to analyze the drop issues * teSummary.pl --IMSI exception --cells contributor --exceptions ==>drop --IMSI * except_time.pl --How often --Inline with peak traffic ? -- Spike or no spike *except.pl *except_cell.pl $ cat 2010-10-14_1026_Rnc04_AllMod_4485.log | teSummary.pl To filter the exceptions based on the certain cell use the following commands $ cat 2010-10-14_1038_Rnc04_AllMod_4488.log |grep -i "cellId = 33784"| teSummary.pl >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> (70) All about synchronization - in RBS Ipsyncref - used for network sync. i.e. the TUB sense the clock signal provided via ETMFX board. The RBS has the ntpServerIp address which it uses to lock the clock. The synchronization client (The RBS) decides the rate of NTP requests from the server(RNC). Algorithm uses only the least delay synchronization packets (Adaptive). NTP sync packets carried on FACH channels. 5993_Bedok_OLP> get 1725 101014-16:53:26 10.251.88.75 8.0g RBS_NODE_MODEL_M_1_8 stopfile=/tmp/988 =================================================================================== ============================== 1725 IpSystem=1,IpAccessHostEt=1,IpSyncRef=2 =================================================================================== ============================== IpSyncRefId 2 administrativeState 0 (LOCKED) availabilityStatus 0 (NO_STATUS) cachedIpAddress ntpServerIpAddress 10.251.88.196 operationalState 1 (ENABLED) reservedBy [1] = >>> reservedBy = TransportNetwork=1,Synchronization=1 syncStatus 2 (OK) userLabel IpSyncRef 2 =================================================================================== ============================== Total: 1 MOs Nodesync MO is in RNC and in RBS under IUB MO. Nodesync is carried over FACH. FACH is used for node sync. =================================================================================== ============================== 3204 RncFunction=1,IubLink=Iub_RBS5993,NodeSynch=1 =================================================================================== ============================== NodeSynchId 1 availabilityStatus 0 (NO_STATUS) fixedWindowSizeInit 12 fixedWindowSizeSup 10 maxAllowedIubRtt 500 maxTransportDelay 8500 operationalState 1 (ENABLED) phaseDiffThreshold 50 phaseMeasurement 21406000 sampleIntervalInit 100 sampleIntervalSup 10 20:53:52 slidingWindowSize staticDelay timeStamp 100 0 2010-10-13 userLabel =================================================================================== ============================== In RNC, Rnc08> get . ntpserver 101014-16:42:50 10.251.148.99 8.0g RNC_NODE_MODEL_L_1_62 stopfile=/tmp/32424 =================================================================================== ============================== MO Attribute Value =================================================================================== ============================== ManagedElementData=1 ntpServerAddressPrimary 10.252.75.13 ManagedElementData=1 ntpServerAddressSecondary 10.252.75.14 ManagedElementData=1 ntpServerAddressThird 0.0.0.0 IpAccessHostEt=MS-4-3 ntpServerMode 1 (ENABLED) IpAccessHostEt=ES2-8-2 ntpServerMode 0 (DISABLED) IpAccessHostEt=MS-25-3 ntpServerMode 1 (ENABLED) IpAccessHostEt=ES-2-25-1 ntpServerMode 0 (DISABLED) IpAccessHostEt=MS-12-3 ntpServerMode 1 (ENABLED) IpAccessHostEt=ES2-4-2 ntpServerMode 0 (DISABLED) IpAccessHostEt=ES-1-8-1 ntpServerMode 0 (DISABLED) IpAccessHostEt=ES-2-8-1 ntpServerMode 0 (DISABLED) IpAccessHostEt=MS-12-2 ntpServerMode 0 (DISABLED) IpAccessHostEt=MS-25-2 ntpServerMode 0 (DISABLED) IpAccessHostEt=ES-2-4-1 ntpServerMode 0 (DISABLED) IpAccessHostEt=ES1-25-3 ntpServerMode 1 (ENABLED) IpAccessHostEt=ES-1-4-1 ntpServerMode 0 (DISABLED) IpAccessHostEt=ES-1-25-1 ntpServerMode 0 (DISABLED) IpAccessHostEt=ES1-4-3 ntpServerMode 1 (ENABLED) IpAccessHostEt=MS-25-1 ntpServerMode 0 (DISABLED) ntpServerMode 0 (DISABLED) ntpServerMode 0 (DISABLED) ntpServerMode 0 (DISABLED) ntpServerMode 1 (ENABLED) ntpServerMode 0 (DISABLED) ntpServerMode 0 (DISABLED) ntpServerMode 1 (ENABLED) ntpServerMode 0 (DISABLED) ntpServerMode 0 (DISABLED) ntpServerMode 1 (ENABLED) ntpServerMode 1 (ENABLED) IpAccessHostEt=MS-12-1 IpAccessHostEt=ES1-4-2 IpAccessHostEt=ES1-25-2 IpAccessHostEt=ES1-8-3 IpAccessHostEt=MS-4-1 IpAccessHostEt=ES2-25-2 IpAccessHostEt=ES2-4-3 IpAccessHostEt=ES1-8-2 IpAccessHostEt=MS-4-2 IpAccessHostEt=ES2-25-3 IpAccessHostEt=ES2-8-3 =================================================================================== ============================== Total: 28 MOs There are counters to be checked in case there is synchronization issue using following commands $pmx -m 1 Synchronization pmHDelayVarBest Date: 2010-10-15 Object Synchronization=1 Synchronization=1 Synchronization=1 Counter pmHDelayVarBest10Pct pmHDelayVarBest1Pct pmHDelayVarBest50Pct 00:45 1 0 3 01:00 1 0 4 01:15 1 0 3 01:30 1 0 5 $ipac_ipsyncref | grep roSync To check NTP connectivity for readclock $ntp peerinfo $nssmpinfo all >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> (71) How to check RSTP is working correctly ? You need to check 2 MOs - SwitchPortStp & SwitchStp. Please note that the port connected to the external world will have "ADMIN_EDGE_ON" and the stpRole will not be "Root" instead it will be DESIGNATED. When port stpRole is "DESIGNATED" it means that port is reponsible for forwarding the packets to the outside world. e.g. Rnc08> hget switchstp bridgePriority|rootBridgeId|mode 101014-12:58:11 10.251.148.99 8.0g RNC_NODE_MODEL_L_1_62 stopfile=/tmp/21535 =================================================================================== ============================== MO bridgePriority mode rootBridgeId =================================================================================== ============================== Subrack=MS,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,SwitchStp=1 0 (PRIO_0) 0 (RSTP) 0000-00:13:5E:D3:81:00 Subrack=MS,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,SwitchStp=1 4096 (PRIO_4096) 0 (RSTP) 0000-00:13:5E:D3:81:00 Subrack=ES-2,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,SwitchStp=1 32768 (PRIO_32768) 0 (RSTP) 0000-00:13:5E:D3:81:00 Subrack=ES-2,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,SwitchStp=1 32768 (PRIO_32768) 0 (RSTP) 0000-00:13:5E:D3:81:00 Subrack=ES-1,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,SwitchStp=1 32768 (PRIO_32768) 0 (RSTP) 0000-00:13:5E:D3:81:00 Subrack=ES-1,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,SwitchStp=1 32768 (PRIO_32768) 0 (RSTP) 0000-00:13:5E:D3:81:00 =================================================================================== ============================== Total: 6 MOs Rnc08> hget switchportstp stprole|stpstate|edgeportmode|remoteBridgeId|priority 101014-12:46:57 10.251.148.99 8.0g RNC_NODE_MODEL_L_1_62 stopfile=/tmp/21535 =================================================================================== ============================== MO edgePortMode priority remoteBridgeId stpRole stpState =================================================================================== ============================== Subrack=MS,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=8,Sw itchPortStp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) Subrack=MS,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=1,Sw itchPortStp=1 2 (ADMIN_EDGE_ON) 128 (PRIO_128) 0000-00:13:5E:D3:81:00 1 (DESIGNATED) 5 (FORWARDING) Subrack=MS,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=2,Sw itchPortStp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:13:5E:D3:81:00 1 (DESIGNATED) 5 (FORWARDING) Subrack=MS,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=3,Sw itchPortStp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:13:5E:D3:81:00 1 (DESIGNATED) 5 (FORWARDING) Subrack=MS,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=5,Sw itchPortStp=1 2 (ADMIN_EDGE_ON) 128 (PRIO_128) 0000-00:13:5E:D3:81:00 1 (DESIGNATED) 5 (FORWARDING) Subrack=MS,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=4,Sw itchPortStp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:13:5E:D3:81:00 1 (DESIGNATED) 5 (FORWARDING) Subrack=MS,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=6,Sw itchPortStp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) Subrack=MS,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=7,Sw itchPortStp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) Subrack=MS,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=8,S witchPortStp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) Subrack=MS,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=7,S witchPortStp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) Subrack=MS,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=6,S witchPortStp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) Subrack=MS,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=5,S witchPortStp=1 2 (ADMIN_EDGE_ON) 128 (PRIO_128) 1000-00:13:5E:D3:7F:3E 1 (DESIGNATED) 5 (FORWARDING) Subrack=MS,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=4,S witchPortStp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 1000-00:13:5E:D3:7F:3E 1 (DESIGNATED) 5 (FORWARDING) Subrack=MS,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=3,S witchPortStp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 1000-00:13:5E:D3:7F:3E 1 (DESIGNATED) 5 (FORWARDING) Subrack=MS,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=1,S witchPortStp=1 2 (ADMIN_EDGE_ON) 128 (PRIO_128) 1000-00:13:5E:D3:7F:3E 1 (DESIGNATED) 5 (FORWARDING) Subrack=MS,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=2,S witchPortStp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:13:5E:D3:81:00 0 (ROOT) 5 (FORWARDING) Subrack=ES2,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=2,SwitchPortS tp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 8000-00:13:5E:D3:7F:C5 1 (DESIGNATED) 5 (FORWARDING) Subrack=ES2,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=3,SwitchPortS tp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) Subrack=ES2,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=1,SwitchPortS tp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:13:5E:D3:81:00 0 (ROOT) 5 (FORWARDING) Subrack=ES2,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=8,SwitchPortS tp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) Subrack=ES2,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=6,SwitchPortS tp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) Subrack=ES2,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=7,SwitchPortS tp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) Subrack=ES2,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=4,SwitchPortS tp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) Subrack=ES2,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=5,SwitchPortS tp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) Subrack=ES2,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=6,SwitchPort Stp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) Subrack=ES2,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=3,SwitchPort Stp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) Subrack=ES2,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=8,SwitchPort Stp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) Subrack=ES2,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=5,SwitchPort Stp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) Subrack=ES2,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=2,SwitchPort Stp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 8000-00:13:5E:D3:7F:C5 2 (ALTERNATE) 3 (DISCARDING) Subrack=ES2,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=7,SwitchPort Stp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) Subrack=ES2,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=4,SwitchPort Stp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) Subrack=ES2,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=1,SwitchPort Stp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 1000-00:13:5E:D3:7F:3E 0 (ROOT) 5 (FORWARDING) Subrack=ES1,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=8,SwitchPort Stp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) Subrack=ES1,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=1,SwitchPort Stp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 1000-00:13:5E:D3:7F:3E 0 (ROOT) 5 (FORWARDING) Subrack=ES1,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=2,SwitchPort Stp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 8000-00:13:5E:D3:80:A6 2 (ALTERNATE) 3 (DISCARDING) Subrack=ES1,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=3,SwitchPort Stp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) Subrack=ES1,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=4,SwitchPort Stp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) Subrack=ES1,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=5,SwitchPort Stp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) Subrack=ES1,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=6,SwitchPort Stp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) Subrack=ES1,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=7,SwitchPort Stp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) Subrack=ES1,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=5,SwitchPortS tp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) Subrack=ES- 1,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=6,SwitchPortS tp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) Subrack=ES1,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=8,SwitchPortS tp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) Subrack=ES1,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=1,SwitchPortS tp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:13:5E:D3:81:00 0 (ROOT) 5 (FORWARDING) Subrack=ES1,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=7,SwitchPortS tp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) Subrack=ES1,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=2,SwitchPortS tp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 8000-00:13:5E:D3:80:A6 1 (DESIGNATED) 5 (FORWARDING) Subrack=ES1,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=3,SwitchPortS tp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) Subrack=ES1,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=4,SwitchPortS tp=1 0 (ADMIN_EDGE_OFF) 128 (PRIO_128) 0000-00:00:00:00:00:00 9 (UNKNOWN) 3 (DISCARDING) =================================================================================== ============================== Total: 48 MOs You might get alarm ESL congestion in RNC if there is loop >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> (72) How to see which ports are active and which are passive. To do this use the counters on the ethernetswitchmoduleport MO as shown below. TESTRNC3820> hpget ethernetswitchmoduleport pm 101014-14:45:36 137.58.194.50 8.0h RNC_NODE_MODEL_M_1_29 stopfile=/tmp/12148 ................................................................................... ....... =================================================================================== ============================== MO pmIfInBroadcastPkts pmIfInDiscards pmIfInErrors pmIfInMulticastPkts pmIfInOctetsHi pmIfInOctetsLo pmIfInUcastPkts pmIfOutBroadcastPkts pmIfOutDiscards pmIfOutErrors pmIfOutMulticastPkts pmIfOutOctetsHi pmIfOutOctetsLo pmIfOutUcastPkts =================================================================================== ============================== Subrack=MS,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=MS,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=1 0 0 0 9 0 187254 1 0 0 0 9 0 6627266 62 Subrack=MS,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=2 0 0 0 0 0 0 0 0 0 0 21 0 1344 0 Subrack=MS,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=3 0 0 0 0 0 2737980 25 0 0 0 22 0 67738 0 Subrack=MS,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=MS,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=4 0 0 0 0 0 2483686 23 0 0 0 24 0 73762 0 Subrack=MS,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=MS,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=MS,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=MS,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=MS,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=MS,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=MS,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=4 0 0 0 0 0 0 0 0 0 0 28 0 2134 0 Subrack=MS,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=3 0 0 0 0 0 0 0 0 0 0 31 0 2334 0 Subrack=MS,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=1 0 0 0 18 0 1502 0 0 0 0 16 0 1024 0 Subrack=MS,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=2 0 3 0 34 0 2526 0 0 3 0 0 0 0 0 Subrack=ES-2,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=ES-2,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=ES-2,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=1 0 0 0 37 0 112422 0 0 0 0 0 0 3569826 33 Subrack=ES-2,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=ES-2,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=ES-2,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=ES-2,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=ES-2,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=ES-2,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=ES-2,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=ES-2,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=ES-2,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=ES-2,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=ES-2,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=ES-2,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=ES-2,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=1 0 0 0 49 0 3844 0 0 0 0 0 0 0 0 Subrack=ES-1,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=ES-1,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=1 0 0 0 50 0 3908 0 0 0 0 0 0 0 0 Subrack=ES-1,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=ES-1,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=ES-1,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=ES-1,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=ES-1,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=ES-1,Slot=26,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=ES-1,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=ES-1,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=ES-1,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=ES-1,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=1 0 0 0 57 0 179902 1 0 0 0 0 0 6523540 61 Subrack=ES-1,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=ES-1,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=ES-1,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Subrack=ES-1,Slot=3,PlugInUnit=1,EthernetSwitchModule=1,EthernetSwitchModulePort=4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 =================================================================================== ============================== Total: 48 MOs Added 48 MOs to group: hget_group >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> (73) RPS (Routing path supervision) Basically RPS is for redundancy. We use it towards north bound routers. It works on ICMP echo messsages methodology between RNC and Default router. If true you will have to configure OwnIpAddressActive & OwnIpAddressPassive. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (74) dscp values in RNC $ get . dscp =================================================================================== ============================== MO dscp =================================================================================== ============================== Ip=1 0 =================================================================================== ============================== Total: 1 MOs Added 1 MOs to group: hget_group ... =================================================================================== ============================== MO ntpDscp =================================================================================== ============================== IpAccessHostEt=MS-25-3 46 IpAccessHostEt=MS-4-1 46 IpAccessHostEt=MS-4-2 46 IpAccessHostEt=MS-4-3 46 IpAccessHostEt=ES1-21-2 46 IpAccessHostEt=ES1-25-3 46 IpAccessHostEt=ES-1-21-1 46 IpAccessHostEt=ES-1-4-1 46 IpAccessHostEt=ES-1-25-1 46 IpAccessHostEt=MS-21-2 46 IpAccessHostEt=MS-21-3 46 IpAccessHostEt=ES1-21-3 46 IpAccessHostEt=ES1-25-2 IpAccessHostEt=ES1-4-2 IpAccessHostEt=ES1-4-3 IpAccessHostEt=MS-25-1 IpAccessHostEt=MS-21-1 IpAccessHostEt=MS-25-2 46 46 46 46 46 46 =================================================================================== ============================== Total: 18 MOs Added 18 MOs to group: hget_group . =================================================================================== ============================== MO dscp =================================================================================== ============================== Mtp3bSpItu=1,M3uAssociation=RNC99 40 Mtp3bSpItu=1,M3uAssociation=USGSN0_2 40 Mtp3bSpItu=1,M3uAssociation=USGSN0_1 40 =================================================================================== ============================== Total: 3 MOs Added 3 MOs to group: hget_group =================================================================================== ============================== MO nbapDscp =================================================================================== ============================== RncFunction=1 0 =================================================================================== ============================== Total: 1 MOs Added 1 MOs to group: hget_group ..... =================================================================================== ============================== MO dscpEgress dscpIngress =================================================================================== ============================== TnlCchQosClassProfile=1,TnlQosClass=1 18 18 RabHandling=1,TnlHspaQosClassProfile=0,TnlQosClass=7 18 18 RabHandling=1,TnlHspaQosClassProfile=0,TnlQosClass=12 18 18 0 18 0 0 0 0 0 0 18 18 18 0 18 0 RabHandling=1,TnlHspaQosClassProfile=0,TnlQosClass=4 0 RabHandling=1,TnlHspaQosClassProfile=0,TnlQosClass=15 18 RabHandling=1,TnlHspaQosClassProfile=0,TnlQosClass=1 0 RabHandling=1,TnlHspaQosClassProfile=0,TnlQosClass=14 0 RabHandling=1,TnlHspaQosClassProfile=0,TnlQosClass=6 0 RabHandling=1,TnlHspaQosClassProfile=0,TnlQosClass=3 0 RabHandling=1,TnlHspaQosClassProfile=0,TnlQosClass=0 0 RabHandling=1,TnlHspaQosClassProfile=0,TnlQosClass=2 0 RabHandling=1,TnlHspaQosClassProfile=0,TnlQosClass=9 18 RabHandling=1,TnlHspaQosClassProfile=0,TnlQosClass=11 18 RabHandling=1,TnlHspaQosClassProfile=0,TnlQosClass=8 18 RabHandling=1,TnlHspaQosClassProfile=0,TnlQosClass=13 0 RabHandling=1,TnlHspaQosClassProfile=0,TnlQosClass=5 18 RabHandling=1,TnlHspaQosClassProfile=0,TnlQosClass=10 0 =================================================================================== ============================== >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (75) Accessing CMXB board Telnet to rnc $lhsh 000100 or 000000 depenidng on which CMXB you want to access $iss ==>Auto login to cmxbsh $ cmx> show pmcounters # See the counters same as EthernetSwitchModulePort MO counter as shown in (72) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> (76) ETIPG Sanity commands $ lh etipg appbh info $ lh etipg appbh hwsup full $ lh etipg appdh info $ lh etipg appsh info $ lh etipg appsh oper >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> (77) Throughput troubleshooting First Qualify the problem - Check for stats - Do a valid comparision with stable stats - before/after - Check for configuration change - relate log events with time The path to troubleshoot UE� Uu�RBS�Iub�DCS(RNC)�PDR (SPBHost)�AAL2U �ET board �Ip or AAL5 �......�SGSN (A) Uu troubleshooting -----------------------#throughput is affected by CQI (1 to 30). Depending on CQI scheduler will decide the modulation scheme(QPSK/16QAM/64QAM) #TBS (transport block size)- the bigger the TBS , better the throughput # Enable following trace on TXB board in RBS. lh tx te e trace1 trace2 trace3 HSSR3_SCHED_RBS_SUPER_TRACE # It includes all the traces required for Dl throughput troubleshooting. [2007-06-07 21:20:55.456] 001800/HSSR3_hsflcIdcServer src/flc_f low_ctrl.c:1919 TRACE2:To MEAS: pqidx: 4, avgPeakUuRate: 2057 kbps, avgUuRate: 2057 kbps, txmEstUuRate: 0 bps, caBitRate: 34 kbps, caCreditsRate: 33 kbps, iubDynDelayFlag: 0, iubFrameLossFlag: 0, iubCoeff 100 %, uuGtIub: 1, activePqf: 0, e stUuRate: 411 5kbps, pqIubRate: 0 kbps pqtCoeff: 32768, avgNumbCCPQ: 316 , caUtil: 0, avgCaUtil: 4453, rbsEstUuRate: 0 bps : 0xAC69 #Check to see that the Uu Throughput required is achieved by the UE: avgPeakUuRate and avgUuRate in the trace; #The trace indicates that in the air the UE should be scheduled at about 2.0 Mbps (avgPeakUuRate: 2057 kbps, avgUuRate: 2057 kbps) =~ 2.0Mbit # Collect the log and then use "strace.pl" to parse the file to see the output $cat mylog.log | strace.pl Also there are pmcounters in RBS that you can monitor. This should be used for live nodes. $ pmxh -m 0.25 HsDschResources pmAckR|pmNackRe # pmAck/pmNack ratio should be around 10% $ pmx -m 0.25 HsDschResources pmused|pmnack Date: 2010-10-15 Object Counter 04:30 Sector=1,Carrier=2,HsDschResources=1 pmNackReceived 9137 Sector=1,Carrier=2,HsDschResources=1 pmUsedCqi 0,865,977,1195,1494,1783,2091,2547,2505,2925,2958,2979,3104,3185,3613,3890,4269,427 3,4280,4450,4892,4592,4814,5129,5028,4702,4349,4022,3659,3141,10515 Sector=1,Carrier=2,HsDschResources=1 pmUsedHsPdschCodes 7083,20,489,0,243,34,141,10,71,6,795,107,1,0,0,0 Sector=1,Carrier=2,HsDschResources=1 pmUsedTbs16Qam 0,236,254,732,1215,260,563,177,3442,5031,613,787,561,2302,0,0,0,0,0,0,0,0,0,0,0,0,0 ,0,0,0 Sector=1,Carrier=2,HsDschResources=1 pmUsedTbs64Qam 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 Sector=1,Carrier=2,HsDschResources=1 pmUsedTbsQpsk 30612,17310,11163,15779,5482,7157,3406,1511,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 ,0,0 Sector=2,Carrier=1,HsDschResources=1 pmNackReceived 7270 Sector=2,Carrier=1,HsDschResources=1 pmUsedCqi 0,100,105,87,227,292,301,393,440,357,431,403,641,775,999,1212,1342,1702,2139,2834,3 080,3770,4920,6392,8178,9454,9654,8966,7514,6153,8210 Sector=2,Carrier=1,HsDschResources=1 pmUsedHsPdschCodes 7408,19,214,5,332,77,81,11,79,774,0,0,0,0,0,0 Sector=2,Carrier=1,HsDschResources=1 pmUsedTbs16Qam 0,395,301,1301,2540,1529,829,2017,4418,5290,795,919,651,1845,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0 Sector=2,Carrier=1,HsDschResources=1 pmUsedTbs64Qam 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 Sector=2,Carrier=1,HsDschResources=1 pmUsedTbsQpsk 14737,20072,7920,10477,4758,7591,2650,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 (B) Iub Troubleshooting ------------------------"IubDataStream" is the MO to be checked in this case. Look for following counters in RBS. SPI03 corrosponds to HS data; for HS related throughput issues always check spi03 e.g. pmCapAllocIubHsLimitingRatioSpi03. pmSumNonEmptyUserBuffers pmSumAckedBits pmNoActiveSubFrames pmNoInactiveRequiredSubFrames pmHsDataFramesLost (IubDataStream) pmHsDataFramesReceived (IubDataStream) pmCapAllocIubHsLimitingRatio (IubDataStream) (unit is 0.1%) e.g. pmCapAllocIubHsLimitingRatioSpi03 pmIfOutErrors (EthernetSwitchPort) (C) RNC Troubleshooting -----------------------ETMFX/ETIPG could be faulty. Check pmCounters on EthernetSwitchPort "pmIfOutDiscards", "pmIfOutErrors" etc Find out for the given UE which dc, pdr device is used using "uer" command. Then enable the following traces on dc, pdr devices $lh dc te e bus_send IUB_UP_DL_UE $lh dc te e bus_send MAC_UP_DL_UE # Counters for dcdevice. To check loading of dcdevice. ################################################################################## MO Class Pm Counters ################################################################################## DcDevice pmSamplesMeasuredDcSpLoad pmSumMeasuredDcSpLoad -------------------------------------------------Else, its core network issue. (D) Enabling trace on PDR device $lh pdr te e all GTP_DL_UE $lh pdr te e trace6 SP_MEAS >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > (78) Channel Switching/ Soft Congestion / Idle-----up---->DCH----downswitch->FACH/Idle >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> (79) HSPA setting in RNC and RBS. Always do following to see the config parameters in RBS-$ diff . HS Settings in the RNC ----------------------#set ChannelSwitching bandwidthMarginUl 90 #set ChannelSwitching=1 dlDownswitchBandwidthMargin 80 #If codeThresholdPdu656 is smaller than numHsPdschCodes the 656 bit RLC PDU size will be used for for cat. 7-10 UE, otherwise the 336 bit RLC PDU size is used #Hsdsch=1 codeThresholdPdu656 0 #Hsdsch=1 numHsPdschCodes 5 #Hsdsch=1 numHsScchCodes 4 (Multiplexing, how many users sharing codes in the same TTI interval) #set IubDataStreams maxHsRate 90 #set NodeBFunction supportOf16qam true match ATM descriptor of RBS and class C for HS HS Settings in the RBS ----------------------#set RbsLocalCell maxNumHsPdschCodes 15 #set carrier hspowermargin 0 #set RbsLocalCell dynamicHsPdschCodeAdditionOn true #set NodeBFunction flexibleSchedulerOn true #set Carrier queueSelectAlgorithm 1 #set Carrier cqiAdjustmentOn true #txdevice numHsCodeResources 3 (If EUL not used) match ATM descriptor of RNC and class C for HS >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> (80) Uu counters in RBS $ pmom . uu pmNoUlUuLoadLimitEul (EDchResources) pmOwnUuLoad (EDchResources) pmTotalRotCoverage (EDchResources) pmNoiseFloor (EDchResources) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> (81) Using complete_mom to see system constants Find out the mom file used for the given node - Just moshell to the rnc and you will see MOM file version as below RNC07> lt all 101025-17:56:57 10.251.131.99 8.0h stopfile=/tmp/14447 Checking MOM version...RNC_NODE_MODEL_L_1_57 then download complete mom file from the following site with the URL as follows (Just add _COMPLETE.xml.gz) http://utran01.au.ao.ericsson.se/dlmom/dlmom.php? file=RNC_NODE_MODEL_L_1_57_COMPLETE.xml.gz http://utran01.au.ao.ericsson.se/dlmom/dlmom.php? file=RBS_NODE_MODEL_M_1_8_COMPLETE.xml.gz to moshell using complete mom put this file in $moshell -v use_complete_mom=1 rnc03 RNC03> lt all 101025-18:01:33 10.251.131.67 8.0h stopfile=/tmp/14565 Checking MOM version...RNC_NODE_MODEL_L_1_57 Using MOM version: RNC_NODE_MODEL_L_1_57_COMPLETE Parsing MOM: /opt/utran/moshell/jarxml/RNC_NODE_MODEL_L_1_57_COMPLETE.xml.gz ................... ..........................................................................Done. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> (82) PLM info 670 Identify -------a) stdr - not allocated cc & dc devices b) lh ranampmp sma -all # check active mp c) lhsh 000700 drhcendh cc & lhsh 000700 drhcendh dc d) All the CC and the DC SPs should be in state [linkEstablished synced allocatedSp allocatedRh] If any SP shows the �pendingRhRestart� flag, then the fault is present Recovery --------$lh ranapmp sma -all # Check which ranapmp is active $lhsh 000700 pm_pginfo -a # run on active mp only and get the pid for "RncLmCenDrh" $lhsh 000700 pm_kill "0x00020252" The process should restart automatically again. Verify it by using $lhsh 000700 pm_pginfo -a Also chek lgv 1h / lgs 1h to see the process has restarted completely. It should have been came up. Then check if the issue still exists or not. Verification -----------a) stdr b) lhsh 000700 drhcendh cc & lhsh 000700 drhcendh dc c) verify distribution of cc device is uniform using lh mod drh_ccrh_topdata look "nrofcell" in the printout. If not evenly distributed then lock unlock the cells on the heavily loaded module. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> (83) How to recover license file from the license server 3944_GlamourVille> acc licensing updateLicenseKeyFile >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
