-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathUsersGuide.tex
More file actions
executable file
·5836 lines (4881 loc) · 245 KB
/
UsersGuide.tex
File metadata and controls
executable file
·5836 lines (4881 loc) · 245 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
\documentclass[12pt]{article}
\usepackage{epsfig}
\usepackage{comment}
\setlength{\textwidth}{6.75in}
\setlength{\textheight}{7.5in}
\newsavebox{\savepar}
\newenvironment{boxit}{
\begin{minipage}[b]{1in}}
{\end{minipage}\fbox{\usebox{\savepar}}}
\setlength{\hoffset}{0.0in}
\setlength{\voffset}{0.0in}
\setlength{\marginparsep}{0.0in}
\setlength{\headheight}{0.0in}
\setlength{\headsep}{0.0in}
\setlength{\topmargin}{0.5in}
\setlength{\textheight}{8.5in}
\setlength{\oddsidemargin}{-0.1in}
\setlength{\evensidemargin}{-0.1in}
\setlength{\marginparwidth}{0.0in}
\setlength{\textwidth}{6.5in}
\setcounter{secnumdepth}{4}
\usepackage[htt]{hyphenat}
\begin{document}
%\def\decaysto{\vbox{
% \hbox{\raise.5ex\hbox{\vrule height10pt}
% \kern-.4025em{$\rightarrow$}}}\kern0.8ex}
\begin{center}
{\Large{\bf CMS Pixel Online Software and Calibrations}}
\end{center}
\begin{center}
\vskip 0.5cm
Anders Ryd ({\it anders.ryd@cornell.edu})\\
Steve Stroiney ({\it srs63@lepp.cornell.edu})\\
Souvik Das ({\it sd259@cornell.edu}) \\
Karl Ecklund ({\it Karl.Ecklund@cern.ch})\\
Josh Thompson ({\it joshua.thompson@cern.ch}) \\
Ben Kreis ({\it benjaminkreis@gmail.com})\\
% (If you make contributions to this document, please add your name above.)\\
\today \\
\end{center}
\begin{abstract}
Pixel Online Software (POS) is the online data acquisition software for the
CMS Pixel Detector installed in 2008. The POS orchestrates its operation,
calibration and monitoring.
This Note describes the structure and installation of this software.
It also documents the algorithms used to calibrate the detector. This document currently
serves both as a Reference Manual for and a User's Guide to POS.
\end{abstract}
\newpage
\tableofcontents
\newpage
\section{Introduction}
This note describes the design and implementation of
Pixel Online Software (POS). POS is used for controlling and calibrating the CMS pixel
detector that was installed in 2008. In particular, it performs the following functions:
\begin{itemize}
\item Configuring the detector
\item Performing online calibrations
\item Analyzing calibration data in online farm (CMSSW)
\item Monitoring the detector during data taking
\end{itemize}
Other information about
Pixel Online Software is available at the pixel online
software wiki pages~\cite{poswiki}.
\section{Pixel DAQ System}
The intention of this section it to give an overview of
the CMS pixel DAQ system. For more details the reader
can follow the references given in the text. But the
introduction given here should be sufficient to understand
the main goals of POS.
\subsection{Overview of DAQ components}
The CMS pixel DAQ system consists of a number of
components as illustrated in Fig.~\ref{fig:daqcomponents}.
This figure is specific to the FPix detector, but the
modification for the BPix are minor and not really
relevant to the discussion here unless otherwise
mentioned. Starting from the detector itself we have the
Read Out Chip (ROC)~\cite{ROC} and the Token Bit Manager
(TBM)~\cite{TBM}. Some properties of the ROC are described
below as needed to understand the calibrations. The
ROC reads out 4,160 pixels and the TBM coordinates
the communication with a group of 8 to 24 ROCs.
The TBM is electrically connected to the portcard via
extension cables $\approx 2$ feet in length.
The portcard receives and sends optical signals to the
Front-End Controller (FEC) and Front-End Driver (FED)
respectively. The FEC is used to program the TBM
and ROCs and the data read out from the detector is
send to the FED to be digitized. The portcard is
home to several discrete components. We have the
Digital Opto-Hybrid (DOH) that receives data from
the FEC and the Analog Opto-Hybrid (AOH) that
transmits data to the FED. On the portcard we
have in addition the (t)PLL, Delay25, and gatekeeper
chips.
Of particular interest is the Delay25 chip. The communication
to the portcard via the DOH is done at 40MHz on a serial
line. We have a clock line and the data line (and return clock
and return data). In order to have this communication
working the timing between the clock and data lines must
be right. The purpose of the Delay25 chip is to adjust
delays to make this communication work. The setting of the
parameters on the Delay25 chip as well as the other components
on the portcard is done using the I2C protocol from the
CCU. The CCU is again controlled from a Tracker FEC (TkFEC).
\begin{figure}[h]
\begin{center}
\includegraphics[width=0.99\textwidth]{2860506-001}
\end{center}
\caption{The main components in the CMS pixel DAQ system. }
\label{fig:daqcomponents}
\end{figure}
\subsubsection{The Read Out Chip}
Each Read Out Chip (ROC)~\cite{ROC}, illustrated in Fig.~\ref{fig:ROC}, collates data from 52 $\times$ 80 = 4,160 pixels
and contains about 1.3 million transistors.
It amplifies and zero suppresses data using 4 trim bits
that specify a threshold for every pixel. The chip buffers hit data
until the trigger decision arrives. It is developed at PSI and manufactured by IBM~\cite{Barbero:2004}.
\begin{figure}[h]
\begin{center}
\includegraphics[width=0.45\textwidth]{ROC.pdf}
\end{center}
\caption{The Read Out Chip of the CMS Pixel Detector.}
\label{fig:ROC}
\end{figure}
For pixel hits to be read out, the following consecutive actions must take place.
\begin{itemize}
\item The accumulated charge in any of the pixels of the double column must exceed
a (programmable) threshold (trim-input to comparator). Then the corresponding time-stamp
(bunch crossing number) is written into the time-stamp buffer presently pointed at, and
the analog signals and the pixel addresses of all hit pixels are written into the next
free data buffers. This hit-recording into the time stamp and data buffers runs autonomously
and asynchronously in each double column of the chip, independently of the bunch crossing
clock. The recorded hit information must be kept in the buffers during the
latency time of the L1-trigger.
\item Hits in the column are validated by an external L1-trigger, by comparing their
corresponding time stamp with a counter running behind the bunch crossing counter by the trigger
delay, else the hits are cleared. If the hits in the column are validated, the value of a 4-bit
trigger counter is latched into the column periphery. The column is frozen and cannot record further
until reset after the readout of the triggered hits. Other (untriggered) columns remain active.
\item All frozen columns with the latched value of the trigger counter equal to the present value
of the token counter are set to the readout mode. Directly afterwards, the trigger counter is
incremented. The readout process starts when the token bit enters the chip. After a three bit chip
header is sent, each consecutive double column which is in the readout mode is read out and reset.
Just before the token leaves the chip, the token counter is incremented. A new token is sent
for each trigger and only data belonging to that trigger will be sent onto the readout bus,
even if more triggers arrive between a trigger and the corresponding token.
\end{itemize}
\subsubsection{The Token Bit Manager}
A Token Bit Manager (TBM) controls groups of 8 to 24 ROCs.
It distributes the clock and trigger signals.
It serializes analog readout using a token bit
passed from ROC to ROC. It is physically mounted next to the ROCs.
The TBM is developed at Rutgers.
\begin{figure}[h]
\begin{center}
\includegraphics[width=0.45\textwidth]{TBM.pdf}
\end{center}
\caption{A schematic of how the Token Bit Manager reads out data from the ROCs.}
\label{fig:TBM}
\end{figure}
The principal functions of the TBM are the following.
\begin{itemize}
\item It controls the readout of the ROCs by initiating a token pass for each incoming
L1-trigger.
\item On each token pass, it writes a header and a trailer word to the data stream.
\item The header contains an 8-bit event number. The trailer contains an 8-bit error status.
These are transmitted in 2-bit analog encoded digital signals.
\item It distributes the L1-trigger and LHC Clock to the ROCs.
\item Each arriving L1-trigger is placed on a 32-deep stack awaiting its associated token pass.
Normally the stack is empty but is needed to accommodate high burst rates due to either noise,
high track density or trigger bursts. Only the first 16 triggers placed on the stack are passed
to the ROCs, and have an associated token pass. All others are marked in the trailer as a ``No
Token Pass" event. This is done to keep the DAQ synchronized, while speeding up the clearing of
the stack.
\end{itemize}
The TBM receives 4 control signals: A 40 MHz clock synchronized to the CMS beam crossing, an active
low reset signal (which resets all control registers to an initial state), a serial line for
configuration settings, and a serial encoded trigger signal.
\subsubsection{The Pixel FED}
The Pixel Front End Driver (FED) is a VME module
with 36 optical inputs. It sits in the counting room and receives the analog output
of the ROCs via an optical fiber from the center of the CMS detector.
In order for the FED to decode and digitize this data, it has to have address levels
well calibrated, and this is one of the several tasks performed by the Pixel Online Software.
Internal timings of the FED also have to be calibrated within errors of a few nanoseconds,
and this is also accomplished using the POS. The output of the FED
is dispatched, in a binary format readable to the entire CMS experiment, via an S-Link.
\begin{figure}[h]
\begin{center}
\includegraphics[width=0.45\textwidth]{pFED.pdf}
\end{center}
\caption{A schematic of the pixel FED.}
\label{fig:pFED}
\end{figure}
\subsubsection{The Pixel FEC}
The Pixel Front End Controller (FEC) sends triggers, clocks
and programming data to the ROCs. For the pixel detector,
we use a one-way `fast' I2C protocol to send this data.
We need to send 1 byte of data per pixel, or 66 MB of data
for the entire detector. For this, we use a 40 MHz serial line,
and this calls for timing calibrations in order to enable downloads.
\begin{figure}[h]
\begin{center}
\includegraphics[width=0.45\textwidth]{pFEC.pdf}
\end{center}
\caption{A schematic of the pixel FEC.}
\label{fig:pFEC}
\end{figure}
\subsection{Installation at P5}
Table~\ref{tab:P5PCs} lists the online PCs that we have at P5
for Pixel Online Software. Table~\ref{tab:FpixFED} lists the FED boards
and their locations in the VME crate within the CMS Counting Room. The slots
and the VME addresses are also provided. Similarly, we list the FEC boards
and their locations in Table~\ref{tab:FpixFEC}. The CCU connections in the Counting
Room are shown in Table~\ref{tab:FpixCCU}.
\begin{table}
\centering
\caption{Pixel online PCs at P5}
\label{tab:P5PCs}
\resizebox{\textwidth}{!}{
\begin{tabular}{lcccccccc}
\hline
\hline
Slot in rack & Node name & Front & Label & Size & Function & Comments & OS \\
\hline
- & vmepcs2b18-17 & Pixel & - & ?U & Histoviewer & & SLC4 \\
19 & vmepcs2b18-16 & Pixel & 1/10 & 1U & VME/S1G01i & BPix FECs & SLC4 \\
17 & vmepcs2b18-15 & Pixel & 10/10 & 1U & VME/S1G01e & FPix FECs & SLC4 \\
16 & vmepcs2b18-14 & Pixel & 9/10 & 1U & VME/S1G04e & BPix FEDs do & SLC4 \\
14/15 & vmepcs2b18-13 & Pixel & 8/10 & 2U & VME/S1G04i & BPix FEDs up & SLC4 \\
13 & vmepcs2b18-12 & Pixel & 7/10 & 1U & VME/S1G03i & FPix FEDs & SLC4 \\
12 & vmepcs2b18-11 & Pixel & 6/10 & 1U & PixelSuperFPix & FPix Supv. & SLC4 \\
11 & vmepcs2b18-10 & Pixel & 5/10 & 1U & PixelSuperBPix & BPix Supv. & SLC4 \\
10 & vmepcs2b18-09 & Pixel & 4/10 & 1U & DCS & Fpix daq & Win \\
9 & vmepcs2b18-08 & Pixel & 3/10 & 1U & DCS/Siemens & & Win \\
8 & vmepcs2b18-07 & Pixel & 2/10 & 1U & DCS/CAEN & & Win \\
& vmepcs2b16-10 & & & & TTC+LTC Supv. & TTC+LTC & SLC4 \\
& cmsrc-pixel & & & & L1 FM & Pixel FM & SLC4 \\
&fmmpc-s1d12-08 & & & & FMM PC & FMM PC & SLC4 \\
& cmspsx & & & & PSX server & PSX server & SLC4 \\
& srv-c2c02-05 & & & & DB server & & SLC4 \\
& srv-c2c02-06 & & & & DB server & & SLC4 \\
\hline
\hline
\end{tabular}
}
\end{table}
\begin{small}
\begin{table}
\centering
\caption{FED connections for FPIX}
\label{tab:FpixFED}
\resizebox{\textwidth}{!}{
\begin{tabular}{llcccclc}
\hline
\hline
Rack & Crate & Slot & VME addr. & Channels & FED id & Name(Official) & Name(Construction) \\
\hline
S1G03 & upper & 6 & 0x13000000 & 13-24 & 33 & BpO\_D(1,2)\_BLD(1,2,3) & HC+Z1 1.1 2.1\\
S1G03 & upper & 6 & 0x13000000 & 1-12 & 33 & BpO\_D(1,2)\_BLD(4,5,6) & HC+Z1 1.2 2.2\\
S1G03 & upper & 7 & 0x14000000 & 13-24 & 34 & Bp0\_D(1,2)\_BLD(7,8,9) & HC+Z1 1.3 2.3\\
S1G03 & upper & 7 & 0x14000000 & 1-12 & 34 & Bp0\_D(1,2)\_BLD(10,11,12) & HC+Z1 1.4 2.4\\
\hline
S1G03 & upper & 5 & 0x12000000 & 1-12 & 32 & BpI\_D(1,2)\_BLD(1,2,3) & HC+Z2 1.4 2.4\\
S1G03 & upper & 5 & 0x12000000 & 13-24 & 32 & BpI\_D(1,2)\_BLD(4,5,6) & HC+Z2 1.3 2.3\\
S1G03 & upper & 8 & 0x15000000 & 1-12 & 35 & BpI\_D(1,2)\_BLD(7,8,9) & HC+Z2 1.2 2.2\\
S1G03 & upper & 8 & 0x15000000 & 13-24 & 35 & BpI\_D(1,2)\_BLD(10,11,12) & HC+Z2 1.1 2.1\\
\hline
S1G03 & upper & 11 & 0x17000000 & 13-24 & 37 & BmI\_D(1,2)\_BLD(1,2,3) & HC-Z1 1.1 2.1\\
S1G03 & upper & 11 & 0x17000000 & 1-12 & 37 & BmI\_D(1,2)\_BLD(4,5,6) & HC-Z1 1.2 2.2\\
S1G03 & upper & 12 & 0x18000000 & 13-24 & 38 & BmI\_D(1,2)\_BLD(7,8,9) & HC-Z1 1.3 2.3\\
S1G03 & upper & 12 & 0x18000000 & 1-12 & 38 & BmI\_D(1,2)\_BLD(10,11,12) & HC-Z1 1.4 2.4\\
\hline
S1G03 & upper & 10 & 0x16000000 & 1-12 & 36 & BmO\_D(1,2)\_BLD(1,2,3) & HC-Z2 1.4 2.4\\
S1G03 & upper & 10 & 0x16000000 & 13-24 & 36 & BmO\_D(1,2)\_BLD(4,5,6) & HC-Z2 1.3 2.3\\
S1G03 & upper & 13 & 0x19000000 & 1-12 & 39 & BmO\_D(1,2)\_BLD(7,8,9) & HC-Z2 1.2 2.2\\
S1G03 & upper & 13 & 0x19000000 & 13-24 & 39 & BmO\_D(1,2)\_BLD(10,11,12) & HC-Z2 1.1 2.1\\
\hline
\hline
\end{tabular}
}
\end{table}
\end{small}
\begin{small}
\begin{table}
\centering
\caption{FEC connections for FPIX}
\label{tab:FpixFEC}
\resizebox{\textwidth}{!}{
\begin{tabular}{llllclc}
\hline
\hline
Rack & Crate & Slot & VME addr. & mFEC & Name(Official) & Name(Construction) \\
\hline
S1G01 & middle & 5 & 0x28000000 & 3 & BpO\_D(1,2)\_BLD(1,2,3) & HC+Z1 1.1 2.1\\
S1G01 & middle & 5 & 0x28000000 & 4 & BpO\_D(1,2)\_BLD(4,5,6) & HC+Z1 1.2 2.2\\
S1G01 & middle & 5 & 0x28000000 & 5 & Bp0\_D(1,2)\_BLD(7,8,9) & HC+Z1 1.3 2.3\\
S1G01 & middle & 5 & 0x28000000 & 6 & Bp0\_D(1,2)\_BLD(10,11,12) & HC+Z1 1.4 2.4\\
\hline
S1G01 & middle & 5 & 0x28000000 & 8 & BpI\_D(1,2)\_BLD(1,2,3) & HC+Z2 1.4 2.4\\
S1G01 & middle & 5 & 0x28000000 & 7 & BpI\_D(1,2)\_BLD(4,5,6) & HC+Z2 1.3 2.3\\
S1G01 & middle & 5 & 0x28000000 & 2 & BpI\_D(1,2)\_BLD(7,8,9) & HC+Z2 1.2 2.2\\
S1G01 & middle & 5 & 0x28000000 & 1 & BpI\_D(1,2)\_BLD(10,11,12) & HC+Z2 1.1 2.1\\
\hline
S1G01 & middle & 10 & 0x50000000 & 3 & BmI\_D(1,2)\_BLD(1,2,3) & HC-Z1 1.1 2.1\\
S1G01 & middle & 10 & 0x50000000 & 4 & BmI\_D(1,2)\_BLD(4,5,6) & HC-Z1 1.2 2.2\\
S1G01 & middle & 10 & 0x50000000 & 5 & BmI\_D(1,2)\_BLD(7,8,9) & HC-Z1 1.3 2.3\\
S1G01 & middle & 10 & 0x50000000 & 6 & BmI\_D(1,2)\_BLD(10,11,12) & HC-Z1 1.4 2.4\\
\hline
S1G01 & middle & 10 & 0x50000000 & 8 & BmO\_D(1,2)\_BLD(1,2,3) & HC-Z2 1.4 2.4\\
S1G01 & middle & 10 & 0x50000000 & 7 & BmO\_D(1,2)\_BLD(4,5,6) & HC-Z2 1.3 2.3\\
S1G01 & middle & 10 & 0x50000000 & 2 & BmO\_D(1,2)\_BLD(7,8,9) & HC-Z2 1.2 2.2\\
S1G01 & middle & 10 & 0x50000000 & 1 & BmO\_D(1,2)\_BLD(10,11,12) & HC-Z2 1.1 2.1\\
\hline
\hline
\end{tabular}
}
\end{table}
\end{small}
\begin{small}
\begin{table}
\centering
\caption{CCU connections for FPIX}
\label{tab:FpixCCU}
\begin{tabular}{llcccc}
\hline
\hline
Rack & Crate & Slot & mFEC & Name(Official) & Name(Construction) \\
\hline
S1G01 & middle & 18 & 8 & BpI & HC+Z2 \\
S1G01 & middle & 18 & 7 & BpO & HC+Z1 \\
S1G01 & middle & 18 & 6 & Bm0 & HC-Z2 \\
S1G01 & middle & 18 & 5 & BmI & HC-Z1 \\
\hline
\hline
\end{tabular}
\end{table}
\end{small}
\section{Pixel Online Software Overview}
\label{sect:overview}
The Pixel Online Software is based on the XDAQ toolkit and is a suite of applications. The
different XDAQ based components are shown in green in
Fig.~\ref{fig:components}. The top level application is the
PixelSupervisor. This application is responsible for the overall
coordination of the pixel DAQ. The PixelSupervisor talks to the
supervisors that directly control the hardware. For example
we have the PixelFECSupervisor that provides the interface to the
pixel FECs. Similarly the PixelFEDSupervisor controls FEDs.
In production at P5, there are multiple instances of the PixelFECSupervisor and
PixelFEDSupervisor; one per VME crate\footnote{The strip tracker
uses a design where there is one supervisor per VME board.}.
The PixelTKFECSupervisor controls the tracker FEC hardware. The
pixel system uses the tracker FEC hardware slow I2C to
initialize the fast I2C used for the download of most
configuration data.
The PixelTTCSupervisor controls the pixel TTC module used for trigger
and timing. Among other things the TTC module is used during
calibrations to generate triggers. In modern releases of the software,
the PixelTTCSupervisor has been deprecated in favor of the
TTCciControl, which is a standard package maintained by the TTC group.
The PixelLTCSupervisor is used for the local trigger control.
The various supervisors run as independent processes, or even on
different computers. Therefore, in order to communicate with each
other they must exchange messages on the network. This is done using
the SOAP protocol.
The Level 1 function manager (L1FM or FM) for the pixels is the
interface the pixel system has to the global run control (RCMS for
Run Control and Monitoring System). The FM is a java application.
It responds to requests from RCMS to change states in the
state diagram that describe the state of the DAQ system. This
state diagram is shown in Fig.~\ref{fig:l1fm}. The pixel
FM is a relatively thin layer that basically just passes the state
changes on to the PixelSupervisor.
\begin{figure}
\begin{center}
\includegraphics[width=0.99\textwidth]{POScomponents.pdf}
\end{center}
\caption{The different applications that compose the Pixel Online Software.}
\label{fig:components}
\end{figure}
\section{Package structure} \label{sect:swcomponets}
In the Pixel Online Software the code is distributed
among a number of packages. These packages are listed
here.
\begin{itemize}
\item PixelCalibrations
%\item PixelCalibrationInterface
\item CalibFormats/SiPixelObjects
\item PixelConfigDBInterface
%\item PixelDCSSupervisor
\item PixelDCSInterface
\item PixelFECInterface
\item PixelFECSupervisor
\item PixelFEDInterface
\item PixelFEDSupervisor
\item PixelFunctionManager
\item PixelLTCSupervisor
\item PixelSupervisor
\item PixelTKFECSupervisor
\item PixelTTCSupervisor\footnote{In use through tag {\tt POS\_3\_1\_2}; deprecated starting in {\tt POS\_3\_2\_0}.}
\item PixelUtilities
\end{itemize}
The package dependency tree is shown in Fig~\ref{fig:dependencies}.
The supervisor applications are at the top and depend on the
packages below. We should make sure that the dependencies
form a tree and not contain loops.
\begin{figure}
\begin{center}
\includegraphics[width=0.99\textwidth]{package_dep.pdf}
\end{center}
\caption{The dependencies among the packages are indicated here.
At the top are Supervisor applications. }
\label{fig:dependencies}
\end{figure}
\subsection{Pixel Function Manager}
\label{sec:l1fm}
The Pixel Function Manager (the Level 1 Function Manager) acts as an interface between
Run Control (the Level 0 Function Manager) and POS. The Pixel Function Manager is a Java
application. It implements the state machine of
CMS~\cite{statemachine}. The Function Manager interacts
with the PixelSupervisor to carry out the
different tasks needed in state transitions of the
run control.
\begin{figure}
\begin{center}
\includegraphics[width=0.99\textwidth]{l1fm_states.png}
\end{center}
\caption{The CMS finite state machine definition. Figure taken from Ref.~\cite{statemachine}.}
\label{fig:l1fm}
\end{figure}
\subsubsection{FSM Implementation Status}
The CMS Run Control finite state machine (FSM) is shown in
Fig.~\ref{fig:l1fm}. This model is implemented in the Pixel Function Manager,
and also in the PixelSupervisor. The other
Supervisors implement portions of this model as required.
The FSM depicted in the figure is completely implemented in
the PixelSupervisor. The PixelSupervisor does
include transitions from any state into the ''Error'' state, and then
allows a ``Recover'' transition that returns the FSM to the ``Halted''
state.
\subsubsection{Control of the L1 Function Manager}
During global running, the L1 Function Manager (L1FM) is driven by the L0FM, which is
operated by central DAQ. The pixel user should not intervene via the
L1FM GUI, except to check its status.
During local running, the L1FM must be created via its GUI. State
transitions of the FSM can then be driven via the L1FM GUI. In
general, we drive the ``Initialize'' state transition via the GUI,
then drive subsequent state transitions directly from the
PixelSupervisor. However, in principle all transitions can be
initiated from the L1FM GUI\footnote{In practice, this is only useful for simple tests of the L1FM to PixelSupervisor communication.}.
\subsubsection{Outline of L1FM implementation}
The L1FM is created using the Create button in RCMS. This calls the
method of \texttt{PixelFunctionManager.java} called \texttt{
createAction()}.
After creation, transitions of the FM FSM are handled
by methods in {\tt PixelEventHandler.java}. Which method is triggered
depends on the transition, as defined by a table near the top of this
class. The names are fairly logical (for example, ``Initialize''
corresponds to the {\tt initAction} method and ``Configure''
corresponds to the {\tt configureAction} method). Inside each {\tt
fooAction} method are two blocks: one to handle objects of type {\tt
StateEnteredEvent} and one to handle objects of type {\tt
StateNotification}.
When an FSM transition ``foo'' is initiated by the L0FM or L1FM GUI,
we enter the {\tt StateEnteredEvent} block of {\tt fooAction}. This
block then contains the code to send a message to PixelSupervisor,
telling it to proceed with transition ``foo''. The L1FM then does
nothing while PixelSupervisor coordinates the necessary activities in
POS. When the transition is completed by the PixelSupervisor, it
passes a message back to the L1FM\footnote{This is implemented in
PixelSupervisor using the {\tt stateChanged} method of the {\tt
rcmsStateNotifier\_} object.}. This message triggers entry into the {\tt
StateNotification} block of {\tt fooAction}. If PixelSupervisor
reports that the transition was successful, this block moves the L1FM
FSM into the state ``foo''. If PixelSupervisor reports an error, this
block moves the L1FM FSM into state ``Error''. In this way, the L0FM
(central DAQ) learns whether the transition was successful.
\subsection{PixelSupervisor}
The PixelSupervisor is the top level Pixel Online Software application
in the Pixel Online Software. As described above it takes
commands from the function manager. There is one pixel
supervisor for the pixel online system.
\subsubsection{Functions}
The main function of the PixelSupervisor is to coordinate the
activities of the other supervisors, particularly during configuration
(see Sec.~\ref{sec:configuration}) and calibration (see
Sec.~\ref{sect:calib}). It is responsible for updating the
configuration database with new settings obtained by calibrations.
The PixelSupervisor also communicates the state of the pixel
XDAQ software (POS) to the Level 1 Function Manager (Sec.~\ref{sec:l1fm}).
\subsubsection{Interface}
The PixelSupervisor web GUI is an html page, which by default
refreshes every few seconds. It displays information about the current
configuration, or if it is not configured it allows the user to select
a possible configuration from a list and configure the detector using
that configuration.
The PixelSupervisor runs the JobControl Monitor, which is a utility
that periodically sends ``heartbeat'' SOAP messages to the JobControl
processes running on the various machines at P5. The PixelSupervisor
GUI uses the replies from these SOAP messages to display whether any of
the POS XDAQ processes has crashed, or whether any of the JobControl
processes themselves are unresponsive. (Note that we typically only
run JobControl at P5, so this feature is not available elsewhere.)
\subsection{PixelFECSupervisor}
The PixelFECSupervisor supervises one crate of pixel-FEC boards.
PixelFECSupervisor may be run by PixelSupervisor via SOAP messages,
or independently through its own web interface called PixelFECSupervisorGUI.
The PixelFECSupervisor is used to load configuration parameters for the ROCs
from the configuration database and program them into the ROCs. Its web interface
offers two levels of interactivity: A finite state machine
based GUI that issues commands to all pixel-FECs in its crate as the user
steps through the states of the FSM, and a low level GUI that gives
the user control down to individual pixels.
The PixelFECSupervisor has a state machine identical to that of the PixelSupervisor
and follows that of the PixelSupervisor. If its state machine is triggered using its
web interface, care must be taken to return it to the state of the PixelSupervisor
before PixelSupervisor is allowed to regain control of it.
\begin{figure}
\begin{center}
\includegraphics[width=0.49\textwidth]{PixelFECSupervisor_Halted.pdf}
\includegraphics[width=0.49\textwidth]{PixelFECSupervisor_Configured.pdf}
\includegraphics[width=0.49\textwidth]{PixelFECSupervisor_Running.pdf}
\includegraphics[width=0.49\textwidth]{PixelFECSupervisor_Paused.pdf}
\end{center}
\caption{The operational states of the PixelFECSupervisor: Halted, Configured, Running and Paused.}
\label{fig:PixelFECSupervisorStates}
\end{figure}
\subsubsection{Halted State}
This is the initial state of the PixelFECSupervisor. As shown in Fig.~\ref{fig:PixelFECSupervisorStates},
the user is allowed to enter a Global Key and press the ``Configure" button. The Global Key is then used
to retrieve configuration data and download them to the FEC hardware while transitioning the state machine
to the Configured state. The Low Level GUI is not accessible yet as the PixelFECSupervisor has not retrieved
the hardware addresses of the FECs it controls.
\subsubsection{Configured State}
``Halt" and ``Start" are the two state machine input keys that are enabled in this state, as shown in
Fig.~\ref{fig:PixelFECSupervisorStates}. The Low Level GUI displays a list of FECs with their VME
base addresses. Clicking on any opens a new browser window with the Low Level GUI corresponding to that
FEC board.
\subsubsection{Running State}
On pressing the ``Start" button, the PixelFECSupervisor transitions to its Running state. It is
shown in Fig.~\ref{fig:PixelFECSupervisorStates}. ``Pause" and ``Halt" buttons are active. The
Low Level GUI is accessible.
\subsubsection{Paused State}
On pressing the ``Pause" button in the Running state, the PixelFECSupervisor transitions to its
Paused state. It is shown in Fig.~\ref{fig:PixelFECSupervisorStates}. ``Resume" and ``Halt"
state input buttons are active. ``Resume" returns the state machine to its Running state,
and hence executes the FEC's part of the last calibration it was last configured for. ``Halt"
takes the state machine back to its ``Halted State".
\begin{figure}
\begin{center}
\includegraphics[width=0.6\textwidth]{PixelFECSupervisorLowLevelGUI.pdf}
\end{center}
\caption{Low Level GUI for the FEC board at VME base address 0x30000000 in the crate controlled
by the PixelFECSupervisor}
\label{fig:PixelFECSupervisorLowLevelGUI}
\end{figure}
Clicking any hyperlink for the FEC boards under the title ``Low Level Commands" opens a new
browser window for the FEC's Low Level GUI. An example of such a GUI is shown in
Fig.~\ref{fig:PixelFECSupervisorLowLevelGUI}. It allows the user to send TBM commands to
individual TBMs, program-DAC and clear-calibration commands to each Readout Chip, and
program-pixel and calibrate-pixel commands to any pixel of any Readout Chip within its control.
\subsection{PixelFEDSupervisor}
A PixelFEDSupervisor supervises one crate of Front End Driver (FED) boards.
PixelFEDSupervisor may be run by PixelSupervisor via SOAP messages, or independently
through its own web interface. Its web interface offers two levels of interactivity:
A state machine GUI that issues commands to all FEDs in its crate while the user steps
through its state machine states, and a Low Level GUI that allows the user to adjust
various settings on any FED board in the crate.
The PixelFEDSupervisor has a state machine identical to that of the PixelSupervisor and
follows that of the PixelSupervisor. If its state machine is triggered using its web interface,
care must be taken to return it to the state of PixelSupervisor before PixelSupervisor
is allowed to regain control of it.
\begin{figure}
\begin{center}
\includegraphics[width=0.49\textwidth]{PixelFEDSupervisor_Halted.pdf}
\includegraphics[width=0.49\textwidth]{PixelFEDSupervisor_Configured.pdf}
\includegraphics[width=0.49\textwidth]{PixelFEDSupervisor_Running.pdf}
\includegraphics[width=0.49\textwidth]{PixelFEDSupervisor_Paused.pdf}
\end{center}
\caption{The operational states of the PixelFEDSupervisor: Halted, Configured, Running and Paused.}
\label{fig:PixelFEDSupervisorStates}
\end{figure}
\subsubsection{Halted State}
This is the initial state of the PixelFEDSupervisor. As shown in Fig.~\ref{fig:PixelFEDSupervisorStates},
the user is allowed to enter a Global Key, a value for the FED Control
Register, and a value for the FED Mode Register and press the ``Configure" button.
The Global Key is then used to retrieve configuration data and download
them to the FED hardware along with the FED Control and Mode Register
settings while transitioning the state machine
to the Configured state. The Low Level GUI is not accessible yet as the PixelFEDSupervisor has not retrieved the hardware addresses of the FEDs it controls.
\subsubsection{Configured State}
``Halt" and ``Start" are the two state machine input keys that are enabled in this state, as shown in
Fig.~\ref{fig:PixelFEDSupervisorStates}. The Low Level GUI displays a list of FEDs with their VME
base addresses. Clicking on any opens a new browser window with the Low Level GUI corresponding to that
FED board.
\subsubsection{Running State}
On pressing the ``Start" button, the PixelFEDSupervisor transitions to its Running state. It is
shown in Fig.~\ref{fig:PixelFEDSupervisorStates}. ``Pause" and ``Halt" buttons are active. The
Low Level GUI is accessible.
\subsubsection{Paused State}
On pressing the ``Pause" button in the Running state, the PixelFEDSupervisor transitions to its
Paused state. It is shown in Fig.~\ref{fig:PixelFEDSupervisorStates}. ``Resume" and ``Halt"
state input buttons are active. ``Resume" returns the state machine to its Running state,
and hence executes the FED's part of the last calibration it was last configured for. ``Halt"
takes the state machine back to its ``Halted State".
\begin{figure}
\begin{center}
\includegraphics[width=0.6\textwidth]{PixelFEDSupervisorLowLevelGUI.pdf}
\end{center}
\caption{Low Level GUI for the FED board at VME base address 0x1c000000 in the crate controlled
by the PixelFEDSupervisor}
\label{fig:PixelFEDSupervisorLowLevelGUI}
\end{figure}
Clicking any hyperlink for the FED boards under the title ``Low Level Commands" opens a new
browser window for the FED's Low Level GUI. An example of such a GUI is shown in
Fig.~\ref{fig:PixelFECSupervisorLowLevelGUI}. A brief description of the buttons follows.
The ``Reload Firmware" button reloads the firmware on all four
FPGAs of the FED and resets the FED. The ``Reset" button resets the FED.
The user can modify the Capacitor Adjustment, Optical Receiver Input and Output Offsets,
and the Channel Offset DACs for each FED channel. One can also read data from Spy FIFO 1,
2 or 3. If FIFO 1 is chosen, a channel must be specified along with a mode -- Transparent
or Normal. One can also choose to ship the data to screen, in which case it will be decoded,
or shipped to a file, in which case a binary dump of the Spy FIFO data will be placed in the
file. It may also be shipped to the RU Builder. However, since we cannot ensure the
stability of the RU Builder in the current version of POS, we do not recommend this option.
The user can modify the phase and delay settings for any channel on the FED using the
``SetPhasesAndDelays" button.
%\subsection{PixelTTCSupervisor}
\section{Coding practices}
\subsection{Makefile}
The Pixel Online Software is built with a
Makefile located in each (sub)package. The
Makefile builds on the XDAQ tools. This
file should ideally be as short as possible
and use the functionallity in the XDAQ package.
You invoke the Makefile in each package by
doing a `make'. There is also a {\tt clean} target.
In addition to the package level makefiles
there is a top level makefile in the pixel
directory. This makefile allows you to build
all the XDAQ packages using {\tt make Set=pixel}.
You can also clean all packages using
{\tt make Set=pixel clean}.
\subsection{Include files}
Include statements should include the file path
starting from the project. For example you should do
\begin{verbatim}
#include "PixelCalibrations/include/PixelAOHBiasCalibration.h"
#include "CalibFormats/SiPixelObjects/interface/PixelCalibConfiguration.h"
\end{verbatim}
\subsection{CVS tags}
We create `official' tags of the form 'POS\_X\_Y\_Z',
e.g. 'POS\_2\_4\_5'. For every tag created, starting
with POS\_2\_5\_0, an entry should be made in the
file pixel/README that describes briefly the new
features in the tag.
\subsection{Building RPMs}
The building of RPMs should be straightforward. The
following steps are required.
\begin{itemize}
\item Prepare the code, update the README file, and VERSION file and
commit and tag the code. The VERSION file contains the version
of the RPM to build.
\item Invoke `make Set=pixel rpm' to build RPMs for all Pixel Online Software
packages.
\item In the {\tt PixelUtilities} directory, invoke 'buildExternalRPMs.sh'
to build RPMs for DiagSystem, TTCSoftware, and FecSoftwareV3\_0.
At the end of building the external RPMs this script copies
all RPMs to \$BUILD\_HOME/RPM\_X.Y.Z-V, where X, Y, Z, and V are
the major version, minor version, patch, and build version,
respectively.
\end{itemize}
The set of RPMs built can be tested (in the online environment) for
consistency by invoking the command 'rpm --test -Uvh *.rpm' in the directory
where all the RPMs are located.
Old RPMs can be cleaned out of the build area using the command 'make Set=pixel cleanrpm'.
\clearpage
\section{Configuration Data Management}
The next two sections describe a C++ interface
for configuration data management and the
different classes that are used to store this
information. For details of the organizational
details of the database structure, please refer
to the presentation:
\begin{verbatim}
http://indico.cern.ch/conferenceDisplay.py?confId=8768
\end{verbatim}
\section{Configuration Database Interface}
Since early summer 2006, we have used an interface for accessing
configuration data in the online software framework.
The access was originally file based, and has now (starting in 2009) been
supplemented with infrastructure to access an Oracle database. The
interface used is rather generic and the purpose of this document
is to write down the interface so that we can have a
clear separation between the database code and the applications
that use data from the configuration database.
First in Sect.~\ref{sect:cppinterface} we describe the C++ API.
The interface is defined in the class {\tt PixelConfigInterface}.
This interface is fully implemented in the {\tt PixelConfigFile}
implementation based on files.
In Sect.~\ref{sect:cmdline} a simple command line tool is described.
This tool is implemented through the C++ interface and should work
both for the file based and the final data base implementation.
Examples below use a few of the configuration data classes that are used.
% In Sect.~\ref{sect:configclasses} the full
% set of classes that are used to configure and control the pixel
% online software are discussed.
\subsection{C++ Configuration Data Access API}
\label{sect:cppinterface}
The class {\tt PixelConfigInterface} in the package
{\tt PixelConfigDBInterface} defines the interface for
access of configuration data. This interface is intended
to provide type safe access methods for retrieving and
storing configuration data.
\subsubsection{Retrieving data from the database}
The primary method for retrieving the data is
\begin{verbatim}
template <class T>
static void get(T* &pixelObject,
std::string path,
pos::PixelConfigKey key)
\end{verbatim}
The {\tt PixelConfigKey} is just an integer that holds the top
level configuration key to be used.
This interface returns a pointer to the data. The caller is assumed
to take ownership of the data and delete it.
The path is a 'secondary key'. It would allow us to store more than
one object of the same type in a given configuration. (In the file
based implementation this label is used to build the path to where the
file is stored.)
Below are a few examples of using this interface.
\begin{verbatim}
PixelConfigKey theGlobalKey(5);
PixelNameTranslation *theNameTranslation=0;
PixelConfigInterface::get(theNameTranslation, "nametranslation/",
theGlobalKey);
PixelDetectorConfig *theDetectorConfiguration=0;
PixelConfigInterface::get(theDetectorConfiguration, "detconfig/",
theGlobalKey);
PixelFECConfig *theFECConfiguration=0;
PixelConfigInterface::get(theFECConfiguration, "fecconfig/",
theGlobalKey);
\end{verbatim}
These examples show how you extract objects for which there is only one
instance of for the
whole detector configuration. You pass in as the first argument a pointer. The
pointer will return 0 if the object was not successfully retrieved\footnote{We have now tried to improve on this error-handling scheme by throwing a {\tt std::exception} in case there is a failure to retrieve the configuration information. When we attempt to retrieve configuration data in the POS software, we both test for a null pointer and handle any exceptions thrown.}. The
second argument is the label for the object. This is essentially a key that
is used to look up the data. The interface is type safe, i.e., if you specify
a path to an object of the wrong type you will get back a null pointer. The
third argument is the global configuration key. This basically specifies
the versions of all objects used in a given configuration. This key is
implemented as an integer.
Besides objects like the name translation and detector configuration listed
above, there are objects such as trim bits, mask bits, and DAC values
that we need
to access on a finer granularity than for the whole detector. To do this
we use slightly modified arguments
\begin{verbatim}
PixelDACSettings *tempDACs=0;
PixelConfigInterface::get(tempDACs,''pixel/dac/FPix_BpI_D1_BLD1_PNL1'',
theGlobalKey);
\end{verbatim}
where we have added to the path the module name for which we want to
extract the
DAC settings.
As a given application, for example the PixelFECSupervisor, will need to access
DAC settings for many modules, and it is more efficient to extract
the data 'in bulk' from the database, we have also added an interface
that allows extraction of multiple objects at the time
\begin{verbatim}
std::map<std::string, PixelDACSettings*> dacs;
dacs["pixel/dac/FPix_BpI_D1_BLD1_PNL1"]=0;
dacs["pixel/dac/FPix_BpI_D1_BLD1_PNL2"]=0;
PixelConfigInterface::get(dacs, theGlobalKey);
\end{verbatim}
The method will add pointers to the modules; if a module is not
in the configuration a null pointer is returned.
In addition to the access method that uses the configuration key,
you can also retrieve data based on the specific version of the
object
\begin{verbatim}
template <class T>
static void get(T* &pixelObject,
std::string path,
unsigned int version)
\end{verbatim}
There is also a version that retrieves all the objects
\begin{verbatim}
template <class T>
static void get(std::map<std::string, T*> &pixelObjects,
unsigned int version)
\end{verbatim}
In general the access based on the configuration key should be used
in the XDAQ applications.
\subsubsection{Storing data in the database}
To store data in the configuration database use the following
method
\begin{verbatim}
template <class T>
int PixelConfigInterface::put(const T* ,std::string path);
\end{verbatim}
This method will install the data in T* using the specified path.
The new version number is returned by the method.
If you instead have a set of configuration data that needs to be
installed you use the interface
\begin{verbatim}
template <class T>
int PixelConfigInterface::put(std::vector<T*>,
std::string path);
\end{verbatim}
Where a vector of pairs is passed, the second argument in the
pair is the sub path used for the object.
\subsubsection{Configuration keys}
A configuration key consists of a set of objects and their versions.
The simplest form is just to specify a list of the paths and the
versions to use in the key
\begin{verbatim}
static unsigned int makeKey(std::vector<
std::pair<std::string, unsigned int> > versions)
\end{verbatim}
The method returns the new configuration key.
\subsubsection{Alias manipulation}
For configurations aliases you can retrieve the list of defined
aliases using
\begin{verbatim}
static std::vector<std::pair<std::string, unsigned int> > getAliases()
\end{verbatim}
The string is the name of the alias and the unsigned int is the
corresponding key.
A very similar method is
\begin{verbatim}
static std::map<std::string, unsigned int> getAliases_map()
\end{verbatim}
That returns a map between the alias name and the corresponding key.
The method
\begin{verbatim}
static void addAlias(std::string alias, unsigned int key)
\end{verbatim}