-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathchange.log
More file actions
3869 lines (3218 loc) · 167 KB
/
change.log
File metadata and controls
3869 lines (3218 loc) · 167 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
version 4.9.0
2014/10/22
- Support SSL-to-proxy: Robots connecting to HTTP forward proxies
using SSL.
To configure the new feature, the robot must mention an address of a
use()d PGL Proxy object. The PGL Proxy object must have an ssl_wraps
field configured. This tells the robot that the forward proxy is
accepting SSL connections.
When a Robot needs to connect to an SSL origin server while using
SSL-to-proxy, the combination results in two independent "nested"
SSL layers: One from the robot to the proxy and one from the robot
to the origin server. The robot sends an HTTP CONNECT request
(encrypted and, hence, invisible to a 3rd-party observer!) inside
the outer SSL layer to the proxy.
- Support requesting and/or serving on-disk files. Configured via PGL
Content.document_root, similar to an Apache httpd document_root.
Polygraph preloads all regular files from Content.document_root into
RAM, scanning document_root sub-directories recursively. Empty files
and files exceeding 2GB in size are currently not supported. When
configured, Polygraph responds to a request with URI path FOO using
the content preloaded from the document_root/FOO file on disk.
- Quit ASAP after reaching a negative phase goal instead of waiting
for server virtual "cold" phase to become idle and time out.
- Exclude the time spent waiting for the first transaction from the
first phase by adding a virtual "wait" phase before the scheduled
real phases.
The wait phase ends when the first transaction begins and should get
no transaction or connection measurements. Wait phase stats are not
logged because there should not be any, and because logging them
would add noise to test reports.
- Added PGL Agent::http_headers field to specify custom HTTP headers.
PGL constants that look like 'name: value' are now parsed as PGL
MimeHeader objects:
MimeHeader B = 'B: valueB';
MimeHeader C = 'C: valueC';
MyRobot.http_headers = [ 'A: valueA', [B, C]: 30% ];
Agent::http_headers is a new kind of PGL "selector" that, for every
selection, goes through all the listed custom headers and selects
each header with the specified probability. In the above example,
Polygraph will emit requests with headers A, B, and C in 30% of all
agent-generated messages while putting just the header A in the
remaining 70% of all messages. Note that header A is always added,
while headers B and C are either added _together_ (30% of cases) or
not added at all.
If a PGL-configured header clashes with a Polygraph-generated header
(or with another PGL-configured header), then the first added header
wins, and Polygraph logs a warning. PGL-configured headers are added
first so they should always win against the Polygraph-generated
headers such as Date, Host, Accept, and Connection. However, this
feature was not really designed for overwriting Polygraph-generated
headers but for adding new/different ones.
- Start SMP workers, one per local Bench host, if Bench.*_side.hosts
are used. To create N SMP workers on host a.b.c.d, use N a.b.c.d
hosts (the PGL clone operator ** is handy for that).
If the Bench hosts array contains a single local host, no worker
processes are started (i.e., the master process becomes the worker
process). This allows Polygraph to run in "legacy" non-SMP mode in
simple Bench configurations.
Changed --worker ID default to zero. The zero value is used for the
Master process because it may create its own logs that should not
overwrite worker logs.
- Support running each SMP worker on a configurable set of CPU cores.
Also usable in a degenerative case of a single worker (no SMP). Not
supported on systems w/o sched_setaffinity() API, such as FreeBSD.
The feature is configurable via the new PGL BenchSide::cpu_cores
field. Each integer array nested inside cpu_cores represents a list
of CPU cores (i.e., a CPU affinity mask) for a single worker. If a
cpu_cores field is defined, the number of its masks must be either
zero or equal to the number of BenchSide's hosts array elements; any
other number is a fatal configuration error. Empty array is treated
as if no cpu_cores array was configured/defined. We may change this
[rather low-level] configuration approach based on user feedback.
Without cpu_cores configured, Polygraph does not try to restrict
workers CPU affinity.
- When generating random content, support optional "data" or "binary"
format when the new Content.generator field is set to "random_data".
This is handy when the DUT treats the default mostly-ASCII content
specially, but the tester does not want to use cdbs to generate
something other than ASCII content.
Note that new "random_data" format is virtually uncompressible,
unlike ASCII that carries more than 25% "fat". Workloads using SSL
compression or other compression mechanisms may see a substantial
different in performance, even without a DUT.
The default behavior is still to use a subset of printable ASCII
characters. That algorithm can now be explicitly configured by
setting Content.generator to "random_text".
- Re-fetch Kerberos TGT ticket if it is needed, but we failed to fetch
it before. The code already dealt with expired and unusable TGT
cases by re-fetching, but a missing TGT led to a persistent stream
of "(c157) missing Kerberos TGT" errors.
- Better reporting of authentication errors: Report such details as
KDC server address, robot identity, and connection information.
- Better reporting of SSL errors: Minimize the number of logged
Comments when detailing a single error.
- Simplified table(filename) PGL calls and command line options: Do
not require explicit distribution type but get it from the loaded
table instead. Old syntax with an explicit type is still supported.
- Fixed rep.rate evaluation in Phase scripts. Polygraph was returning
request rate instead of the response rate. The two rates are usually
similar, which may explain why the bug went unnoticed for years.
Logged and console-reported stats were not affected.
- Fixed KDC error handling: Polygraph was not switching to the next
configured KDC server on KDC connection failures.
Also, final TGT ticket fetch failure (no retries left) now leads to
the cancellation of all previously queued ticket transactions.
Prior to this change, the queued ticket transaction waited forever
if the TGT transactions kept failing.
- Count HTTP authentication errors once, not twice.
- Avoid segfaults when cleaning up Kerberos state, especially after
many KDC timeouts.
- Prevent InAddress::sockAddr() assertions when deleting IPv6 aliases
on FreeBSD.
- Avoid year-2038 overflows when running 32bit ltrace --objects time.
- Made modern autotools happier and bootstrap.sh safer.
version 4.8.0
2014/05/20
- Tie Robot.req_rate to "baseline" transactions and report just the
baseline transactions on console i-lines, avoiding user surprises
when, for example, enabling authentication doubled perceived request
rate. A baseline transaction delivers the intended resource to the
client and may include individual HTTP CONNECT or authentication
transactions as intermediate steps of a single compound transaction.
This is a significant change, affecting offered load in
workloads using HTTP CONNECT requests and/or HTTP authentication.
Collect and report interval stats for HTTP CONNECT and
"authenticating" transactions. Exclude HTTP CONNECT requests from
"ssl" interval stats so that only transactions actually using SSL
are recorded there. Report an active i-proto line, even if all its
transactions have failed.
Binary log version, some lx object names, and HTML report blobs have
changed to reflect the new statistics collection approach. More
related cleanup work is probably needed.
- Negotiate/Kerberos proxy authentication support, configurable via
PGL KerberosWrap. Both UDP and TCP transports are supported for
communication with KDC. MIT Kerberos v1.9 or newer is required. No
Heimdal support.
- Support a configurable cache for successful DNS lookups via the new
PGL DnsCache type and DnsResolver::cache field. No DNS caching by
default.
- Added PGL zoneDomains(file_name, ip_addresses) function to load
domain names from DNS zone files. Only domains that match given IP
addresses are loaded. Wildcard domains are ignored.
DNS zone file interpretation is implemented using libldns. Libldns
does not allow any leading whitespace in zone files, even on
otherwise empty lines. The library supports $ORIGIN directive.
Tested with libldns v1.6.16. Polygraph tries [too] hard to fix
known ldns.h interface bugs. If libldns is not available, cannot be
fixed, or is not compatible with Polygraph expectations, a
./configure warning is printed and any test calling PGL
zoneDomains() will fail at call time.
- Control SSL compression via the new PGL SslWrap::compression field
which specifies the probability that the agent using the SslWrap
configuration will try to use SSL compression. The agent decision is
sticky. Disabling compressions appears to be required to enable SSL
(AES-NI) hardware acceleration support, at least in some
environments. Note: SSL compression is a messy feature in general,
and Polygraph specifically does not have full control over SSL
compression.
- Better SMP support (more SMP automation work is still needed):
* Added --worker command line option to specify SMP worker ID.
* Support %worker macro in command line option values.
* The local r.n.g. seed value now defaults to worker ID. This
change does not affect old test scripts running new Polygraph
versions because the default worker ID is one, same as the old
local r.n.g. default.
* Added worker() PGL function to return the current worker ID.
* "${worker}" macro support in ips2names, credentials, URL traces.
* pgl2ldif --worker ID command line option.
- Revised detection of important differences reported by cmp-lx.pl
based on real-world regression tests to minimize noise. For example,
very rare (less than 0.1%) errors are no longer considered important
and absolute differences below 1 millisecond are no longer reported
for time measurements done with millisecond precision.
- Infect content by appending Request URI information to the response
body if the newly added PGL Content::inject_object is set to a
"request_uri" string. This helps caches that do not store multiple
copies of identical response bodies, especially in tests using cdbs
with a small number of verbatim entries.
- Accept "A" values for DnsResolver::query_type. They are still the
default.
- Report an active i-proto line, even if all its transactions have
failed.
- Revamped Reporter's baseline (a.k.a. executive stats) scope
formation and use, to better handle cases where client- and
server-side phases differ.
- Support patches that "register" themselves with Polygraph so that
Polygraph binaries can report which patches have been applied.
- Do not enable SSL support if libssl is missing or has no
SSL_connect(), even if OpenSSL headers are present.
- Remove old IPv6 addresses when creating new aliases. Fixed(?)
primary interface address detection on Linux. We currently define
"primary" as the address added to the interface first. Also fixed
IPv6 aliases deletion.
- Fixed handling of concurrent HTTP Negotiate transactions when one
transaction changes the proxy need for authentication after the
other transaction has already started with a different assumption.
- Fixed collection of transaction queue time statistics.
- Fixed default libz (compression support) detection for systems with
AX_CHECK_ZLIB serial 8 macro.
- Fixed StatCycle.cc:82 "0 < cat < count" assertion.
- Avoid division by zero "floating exception" when producing
dynamize()d domains for empty worlds. If we need to produce a new
URL but cannot (due to an empty local world), then repeat from a
non-empty remote world (if possible).
- Fixed dumping of cdbs with "binary" entries. They used to be
truncated.
- Prevent cdbEntries.cc 'e.model' assertion by rejecting misconfigured
Content that has content_db containing "markup" objects but lacks a
may_contain field necessary to generate/pour those objects.
- Fixed computation of histogram percentiles by cml-lx.pl tool.
- Fix segfault in PGL parser on some invalid workloads.
- Install the documentation files during "make install".
- Remove AX_* macros from cfgaux/, use system versions instead.
Autoconf-archive package is now required for bootstrapping.
version 4.7.0
2013/04/19
- Configurable SSL certificates.
A new PGL SslWrap::certificates field allows to configure existing
certificates for Polygraph client and server use. Each certificate
file should contain an unencrypted certificate and a private key in
PEM format. The key may be followed by intermediate certificates to
form a chain (see SSL_CTX_use_certificate_chain_file(3)).
Certificates are assigned to Agents in a round-robin fashion and the
assignment is sticky.
This feature is supported on both client and server sides. If the
same SslWrap object with certificates is used for both Robots and
Servers, they would end up using the same certificates.
- Generated client-side SSL certificates.
Polygraph used to generate server certificates only. A new PGL
SslWrap::generate_certificates boolean field now explicitly controls
whether Polygraph should generate SSL certificates. The field is
honored on both client and server sides. The default behavior has
not changed: certificates are not generated on the client side and
are generated on the server side.
Setting SslWrap::generate_certificates to false allows running
server without any certificates. This is a weird but valid
configuration.
- Collect and report SSL session resumption phase statistics.
"Count" and "rate" phase stats are available for new and resumed SSL
sessions. On client side, both offered (i.e. resumption attempts)
and measured stats are collected. On server side, only measured
stats are collected. In addition to count and rate stats, SSL
session resumption probability is reported.
In the future, we may add offered SSL session stats on the server
side and collect interval stats.
- Content-driven popularity model support.
A new ClientBehavior.pop_model PGL field added. Similar to other
ClientBehavior properties, it allows setting Robot popularity model
when requesting a particular Content type.
- Binary log compression.
Compression is enabled at build time if and only if zlib library is
available. All Polygraph tools support reading both compressed and
uncompressed logs. A compressed log size for a simple workload is
less than 20% of the uncompressed log.
Other than smaller log sizes, no user-visible changes are expected.
In particular, we do not expect log compression to slow Polygraph
down, but more experience is needed to confirm that.
- Inherit previous phase factors at runtime (by default).
If the next phase "begin" factor is not explicitly configured, it is
inherited from the corresponding "end" factor of the previous phase.
Before the change, this inheritance was implemented at test start
time so when a phase would end [by its script] prematurely, the
factor would jump from the last value in the previous phase to the
implicitly set "begin" value in the next phase. Now, the "begin"
value is computed only when the next phase starts, avoiding the
jump.
- Polymorphic min() and max() PGL functions.
Before the change, PGL min() and max() functions converted their
arguments to float for comparison. Hence, the functions failed for
argument types which cannot be converted to float, such as time.
Now, arguments are compared using PGL '<' and '>' operators,
allowing min() and max() functions to be applied to such types as
time and rate.
- More flexible PGL string to int and float conversion.
Before the change, PGL string to int and float conversion functions
did not allow any spaces after the number, which resulted in obscure
errors and inconveniences, especially when dealing with numbers loaded
using PGL system() calls. Now any trailing spaces are ignored.
Note that leading spaces were already ignored before the change.
- Allow unit scales and boolean constants as PGL function and
object names.
Before the change, time and size scales and boolean constants were
not allowed to be PGL function or object names. In particular, this
resulted in parsing errors for workloads using min() PGL function or
scripts using objects like sample.rptm.min. Now PGL parser can
distinguish bare constants from function calls and object names.
- Select Robot HTTP version on startup and report it.
Before the change, Robot HTTP version was selected on session start.
Hence, Robot HTTP version could be changed during a test with
multiple sessions and it could not be reported on startup. Since
sessions are not meant to model different HTTP clients, such
behavior is wrong. Now HTTP version selection for Robots is sticky
and performed early during startup, the same way it is done for
Servers. As a side-effect, Robot HTTP versions are now reported on
startup.
- Stop Robots when all phases are complete.
Polygraph client no longer starts new transactions after completing
the phase schedule and right before exiting. Such transactions could
lead to annoying proxy and server errors.
- Support PGL print() call outside of Phase scripts to assist with
debugging complex workloads and annotating console logs. For now,
output printed outside of scripts does not get to binary logs.
- Add printV() function for PGL scripts to print at a given verbosity
level. The first argument should be an integer specifying the
desired verbosity level. For example:
printV(1, "An important message ", 123);
- Polish the error message when PGL type conversion fails.
- Polish client error message when a Robot fails to load the
configured root certificate.
- Improve large table distribution (e.g. array selectors) support.
Table distributions with up to 10240 entries should be supported
now. Larger distributions may still result in failures.
- Refuse to add directories to cdb.
Before the change, the cdb tool did not check if input file is
really a file. It would try to open and process a directory as if
it was a file, which may succeed on some systems and result in
unexpected DB content. Now cdb checks whether input file is a
regular file and skips it if it is not. If stat(2) is not
available, the check is not enabled and cdb behaves as before.
- Fix handling of CONNECT tunnels requiring multiple CONNECT
transactions. Do not switch to SSL until CONNECT tunnel is fully
established. In particular, this fixes workloads using both HTTPS
(SSL) and proxy NTLM authentication.
- Fix NTLM authentication statistics for compound transactions.
During the first authentication Robots cache the authentication
scheme. In subsequent requests, Robots send appropriate
authentication headers right away in the first request. Before the
change, a compound transaction was created only during the first
authentication resulting in missing compound stats for subsequent
NTLM authentications.
- Fix HTML report generation if a phase name contains spaces.
- Fix segfault for workloads using foreign trace without any Content
objects (an uncommon but valid configuration).
- Fix segfault on test shutdown before warmup stage is complete.
- Fix potential reporter segfault on some truncated logs.
- Fix errors and warnings when building with GCC v4.7.
- Code cleanup and minor changes.
version 4.6.0
2012/05/07
- Add reachedPositiveGoal() PGL script call, propagate positive
goals to servers.
A new reachedPositiveGoal() call allows to stop phases from
runtime watchdog scripts. The effects of the call are the same as
reaching any other positive goal (e.g. Phase duration or
transaction count). If multiple client processes are in use,
calling reachedPositiveGoal() in one client process is not
sufficient - the synchronized phase will not really stop until all
clients are done with it.
Positive goals are now propagated from clients to servers. This
means that when all clients reach positive goals, servers would
also reach a "propagated" positive goal. This was implemented to
propagate reachedPositiveGoal() calls from clients to servers in
the first place (servers do not run watchdog scripts) but it works
for any positive goal. Currently, there is no way to disable this
behavior. In the future, if we find out that there are cases
where servers should resist phase changing on client-side positive
goals, we can probably add a phase.goal.mandatory or similar flag
to give users more control.
- Per-IP explicit port manager (--ports option) for client and
server. Before the change, explicit port manager had a single
port history for all IP addresses. This makes explicit port
manager a poor performer when we use 1000 robots with 1000
different IP addresses. Now it does proper per-IP port tracking.
This change increases Polygraph memory usage for clients and
servers using explicit port managers. Each explicit port manager
for 1024:65535 range uses ~250KB of memory. A workload with 1000
IP addresses would require additional ~250MB of RAM.
- Make retried transactions obey request inter-arrival
distribution and, hence, Robot::req_rate setting.
Transactions may be retried due to several reasons like aborts,
pipeline races, authentication. Before the change, retried
transactions did not obey request inter-arrival distribution.
This resulted in "extra" traffic and higher request rate which
feels wrong and is confusing. Retries are now handled in the same
way as embedded objects: retried transactions are started
immediately but further requests are delayed to obey configured
request rate.
- Interactive figures in HTML reports using jqPlot.
There are two kind of figures now: static and interactive.
Static figures use regular images as before, while newly added
interactive figures offer advanced features:
* resizing
* zooming
* line toggling
By default, all figures are shown in static mode. One can
switch between static and interactive figures using the links in
the top right corner.
A modern browser is required for interactive figures.
Interactive figures are implemented using jQuery, jQuery UI, and
jqPlot Javascript libraries. The libraries are included in Web
Polygraph distribution and are installed into the data directory
(/usr/local/share/polygraph by default). Web Polygraph must be
installed (make install) for reporter to work.
- Other fixes and improvements for figures in HTML report. In
particular, empty ranges warnings from gnuplot are fixed which
makes reporter output much less verbose and more readable in some
cases.
- Enable SSL certificate extensions when making certificates using
OpenSSL x509. According to x509(1ssl) manual page, without the
-extfile option, OpenSSL x509 command will not add extensions to
the certificate.
- Remove OpenSSL x509 -extensions option as not needed. The
section with certificate extensions can be specified in the
OpenSSL configuration file instead.
- Fix client segfault on shutdown before warmup is complete.
- Fix client and server segfault when PGL string array is use()d.
- Fix reporter segfault on some truncated logs, print warning when
log is truncated.
- Fix segfault and assert in ltrace introduced by detailed
content-type stats changes in v4.5.0.
- Fix pgl2ips segfault when there are many hosts or agent types
configured.
- Fix several segfaults and bugs in polymon tool.
- Add help screen for polymon (press 'h' or 'H' key).
- Fix undefined blob reference errors in reporter for one-sided phases.
- Fix build on FreeBSD 7.2.
- Remove unused --tmp_dir option for reporter and comparator.
- Code cleanup and minor changes.
version 4.5.0
2011/11/01
- Add detailed content-type stats. Before the change, only
aggregated size stats were collected for individual content
types. Now, Polygraph collects and reports aggregated stats,
histograms, and traces for content types, including both size
and response time stats.
- Support longer tests. Before the change, Polygraph used 32bit
integers for stats, OIDs, and related code. When running very
long and/or very high-load tests (e.g., a few days at 10Gbps
speeds), 32bit counters could overflow resulting in weird
errors and assertions. Now Polygraph uses 64bit integers for
all counters, OIDs, and related code, avoiding overflows.
Note: Since Polygraphs prefers to keep URL lengths constant,
the length of Polygraph URLs and dynamic domain names have
increased. Polygraph URL path length changed from 42 (e.g.,
/w184f4bb3.78a56c48:00000008/t03/_00000001) to 50 (e.g.,
/w184f4bb3.78a56c48:00000008/t03/_0000000000000001). Dynamic
domain name prefix length changed from 7 (wNNNNNN) to 17
(wNNNNNNNNNNNNNNNN).
- The above changes may result in significant binary log size
increase for some workloads.
- Increase minimum supported log version due to the above
changes.
- Replace MINSTD (described in "Random Number Generators: Good
Ones Are Hard To Find" by Stephen K. Park and Keith W. Miller)
rng with the Unix rand48(). The rand48() rng allows us to
generate 32 random bits as opposed to 31 bit from MINSTD. This
lets us implement 64bit rng by generating two random 32bit
numbers. Besides the rand48() rng demonstrates better results
in dieharder tests.
This change affects random variables generated by Polygraph,
hopefully for the better. For example, we observed more precise
distribution of protocol choices during a regression test using
HTTP and FTP transactions.
- Send Host header in HTTP CONNECT requests. Per RFC 2616, a
client MUST include a Host header field in all HTTP/1.1
request messages. Before the change, Polygraph did not send
Host header in CONNECT requests.
- Fix multiple bugs related to IPv6 alias creation on Linux (at
least). The bugs led to Polygraph assertions and corrupted
IPv6 addresses being created. Now, Polygraph can successfully
create IPv6 aliases on Linux (at least). Currently, in
contrast with IPv4, Polygraph does not delete old IPv6 aliases
before creating new ones.
- Fix IPv6 alias support in the polygraph-aka tool. Now it can
successfully create IPv6 aliases (at least on Linux), e.g.:
polygraph-aka lo '[::3]'.
- Work around a bug on FreeBSD when many IPv6 aliases are
created on a real network interface. Polygraph tried to use
the aliases immediately after creating them and apparently
before they were fully configured. The problem affects at
least FreeBSD 7.2.
- Fix build with OpenSSL libraries with disabled SSLv2 protocol.
In particular, this fixes build on Debian Weezy. Workloads
using SSLv2 will be rejected if Polygraph was built without
SSLv2 support.
- Fix persistent working set ID reporting when loading the
stored set. Before the change, a new ID was generated each
time the set was loaded.
- Cleanup copyright preamble to make Debian folks happier: add
copyright statements where missing, remove copyright from
generated config.h file, update copyright years, replace "(C)"
with "Copyright".
- Remove files specific to the Windows build which has not been
supported for a while.
- Code cleanup and minor changes.
version 4.4.0
2011/03/28
- Content-driven recurrence support: ClientBehavior.recurrence
PGL field added. Similar to other ClientBehavior properties,
it allows setting Robot recurrence for a particular Content
type. For example, a Robot can now be configured to revisit
images more often than movies. The feature is documented at
http://www.web-polygraph.org/docs/userman/content-driven-robots.html.
The object ID (oid) generation and management algorithms have
been changed to support content-driven robot recurrence.
Polygraph clients now have to synchronize a larger number of
smaller distributed oid sets (a.k.a. "worlds"), especially for
workloads with a large number of distinct content types.
As a result of these changes, BHR discrimination has fewer
side-effects now. Before these changes, the BHR discrimination
algorithm affected content type selection. This could make
actual content type distribution and recurrence ratio
different from the configured ones, especially for workloads
with embedded objects. After these changes, the traffic
should better match the PGL configuration.
No significant changes in traffic patterns are expected for
high-performance workloads not using BHR discrimination.
- Added PGL system() function. The function accepts a single PGL
string as an argument, treats that argument as a shell
command, executes the command, and returns the standard output
of the command as a PGL string value.
Outside Phase scripts, the system() function is equivalent to
a back quoted `command` and is executed only once at startup.
Inside Phase scripts, the system() function is executed every
time the script is run, unlike back quotes which are executed
only once at startup, even in a Phase script context
- Taught PGL int() and float() functions to convert PGL strings
to integer and floating-point numbers. Implicit conversion
from strings to numbers are still not allowed.
- Fixed assertion when a container PGL Content type uses an
embedded PGL Content that was not configured yet.
- Fixed layout of tables printed on the console (e.g. "Server
content distributions" and "Phases").
- Code cleanup.
version 4.3.2
2011/04/05
- Fixed assertion when a container PGL Content type uses an
embedded PGL Content that was not configured yet.
- Fix build with OpenSSL libraries with disabled SSLv2 protocol.
In particular, this fixes build on Debian Weezy. Workloads using
SSLv2 will be rejected if Polygraph was built without SSLv2
support.
- Fixed layout of tables printed on the console (e.g. "Server
content distributions" and "Phases").
- Fix persistent working set ID reporting when loading the stored
set. Before the change, a new ID was generated each time the set
was loaded.
- Cleanup copyright preamble to make Debian folks happier: add
copyright where missing, remove copyright from generated config.h
file, update copyright years, replace "(C)" with "Copyright".
- Remove files specific to Windows build which has not been
supported for a while.
version 4.3.1
2011/03/03
- Better support for trace replay: Ignore Robot.origins field
for PGL Robots with foreign-only interest. Before the change,
non-empty Robot.origins field was required, even if it was
unused because the Robot has no interest other than replaying
the foreign trace.
- Better support for trace replay: Do not "warm up" Robots with
foreign-only interest. Before the change, many workloads
containing trace-replaying Robots could get stuck in the first
phase due to an infinite warmup. Now, foreign-only Robots do
not affect the warmup, and the first phase ends as expected.
- Print the corresponding workload file line and number for many
PGL-related warnings and errors.
- Fixed client crash on exit when Robot.accept_content_encodings
PGL field is set.
- Fixed generation of lifecycle-related entity parameters when
the same ObjLifeCycle object is used multiple times in the
workload. The bug resulted in unexpected entity-header (e.g.,
Last-Modified-Time and Expires) changes when the same URL was
revisited.
- Fix statistics recording for aborted transactions. Before the
change, non-basic (e.g. POST) aborted transactions were
recorded in the wrong category (e.g. post). Now, all aborted
transactions are recorded in the aborted category.
- OpenSSL 1.0 support (LP:690599).
- Fixed epoll(2)-related bug which could result in "Socket not
connected" and similar errors.
- Fixed select(2) support in SSL-related workloads.
- Code cleanup.
version 4.3.0
2010/12/10
- Content-driven Robots support: The user may configure Robot
behavior based on Content. Real HTTP client behavior often
depends on web site and its content. Content-driven Robots
help simulate those dependencies.
All for-Robot settings in PGL Content objects are grouped
under the new Content::client_behavior field. A new PGL
ClientBehavior type is added. Assigning PGL Robot to
ClientBehavior and vice versa is supported. Once Server and
Content are selected, Content::client_behavior settings, if
any, overwrite the corresponding Robot fields.
For now, the following three Robot behavior properties can be
specified on per-Content basis: req_types, req_methods, and
ranges.
- Considerably improved the randomness of distribution values
tied to URLs, such as response sizes. The old distributions
could have too many similar values, with too few extreme
values, especially when non-unique worlds (--unique_world no)
were used with multiple PGL Servers and table-based
distributions.
- Linked object blobs from traffic stream tables in report.
- Fixed poll(2)-related bug which could result in "Socket not
connected" or similar errors on platforms without epoll(2)
support such as Mac OS X.
- Multiple world support for distr_test tool: The world_count
option is intended to simulate URL-based value generation in
workloads with multiple PGL Servers, each Server being a "oid
world" for the Robot. By default, world_count is 1, and all
oids are generated from a single world. If world_count is
greater than 1, distr_test generates oids from multiple
worlds. The number of oids generated from each world is
(trials count / worlds count).
- Minor fixes and code cleanup.
version 4.2.2
2010/11/05
- Added custom stats: The user may configure which transactions
are added to custom stats based on HTTP status code.
Configuration is done with Agent.custom_stats_scope PGL field:
custom_stats_scope = ["all", "-2xx", "+30x", "+206"];
If a transaction finishes before getting a reply, or there is
no HTTP status code for another reason, the transaction is not
included in the custom stats.
The following custom stats are collected: custom.rep,
custom.live.rep, custom.projected.rep. Custom stats are
accessible from PGL phase scripts.
- Fixed PGL TblDistr to produce the same results for the same
seed. For example, if a table distribution is used for
response body sizes, the response size would stay the same
when the same URL is visited multiple times.
- Fixed several build and startup problems on Mac OS X.
- Fixed a DummyNet-related build problem on FreeBSD 8.1.
- Fixed IPv6 address parsing in command line arguments.
- Fixed segfault when parsing a PGL array or list with a
malformed item.
- Allow IPv6 addresses in PGL to start with '[:'.
version 4.2.1
2010/10/14
- Transaction lifetime support: Allow the user to limit the
lifetime of client and server transactions via the new PGL
Agent::xact_lifetime field. A transaction aborts when its
configured lifetime expires. Aborted transactions are treated
as errors on client and server sides of the test, for now.
Client transaction lifetime includes both connection slot
waiting time (if any; see Robot::open_conn_lmt) and regular
execution time. Server transaction lifetime includes both think
time (if any; see Agent::xact_think) and regular execution time.
version 4.2.0
2010/10/08
- Allow PGL scripts to monitor in-flight or "live" transactions
so that they can adjust load even when many transactions get
stuck or slow down and do not immediately contribute to regular
postmortem stats. Measure how long live transactions have
been running (live.rep.rptm) and how many response bytes they
have read (live.rep.size). Make these new stats available to
PGL scripts via the regular stats interfaces
(currentSample().live.* and currentPhase().live.*).
- Added projected.rep.* statistics that combines live and
finished transaction stats. Available to PGL scripts only.
- Treat an HTTP CONNECT response as the end of the transaction.
Collect and report stats dedicated to these new CONNECT-only
transactions. These CONNECT transactions are often rather
different from the in-tunnel transactions that follow.
Isolating them also prevents CONNECT overheads from polluting
the first in-tunnel transaction stats.
version 4.1.2
2010/11/06
- Fixed PGL TblDistr to produce the same results for the same
seed. For example, if a table distribution is used for
response body sizes, the response size would stay the same
when the same URL is visited multiple times.
- Fixed several build and startup problems on Mac OS X.
- Fixed a DummyNet-related build problem on FreeBSD 8.1.
- Fixed IPv6 address parsing in command line arguments.
- Fixed segfault when parsing a PGL array or list with a
malformed item.
- Allow IPv6 addresses in PGL to start with '[:'.
version 4.1.1
2010/10/07
- Add dump command to the cdb tool, which stores all verbatim and
link-only objects into individual files. Cdb dump accepts an
optional --as "path/prefix${name}.suffix " parameter. Default
--as value is "./cdb${name}". ${name} macros are substituted with
1,2,3,4... integers.
- Cdb interface has changed. Commands (add, dump, show) are passed
as options with database argument, i.e. cdb --show test.cdb.
- Limit PGL load and population factors value to [0%, 100%] range.
Print error and exit if begin or end factor value is greater than
100%. If a script changes factor to a negative or greater than
100% value, print warning and set the factor to 0%/100% instead.
- Set Recursion Desired (RD) flag for outgoing DNS queries because
some modern resolvers do not accept queries without that flag
and because we do not recursively resolve ourselves.
- Skip already configured network interface aliases when creating
new ones if delete_old_addrs option is false.
- Do not send Content-MD5 header with HTTP 206 (Partial Content)
responses. This fixes segfault and is in line with HTTPbis.
See http://trac.tools.ietf.org/wg/httpbis/trac/ticket/178
- Fix assertion during network interface alias creation.
- Fix errPersistButNoCLen error on client-side when multi-ranges
and persistent connections are enabled. Content-Length is not
required if message uses the media type "multipart/byteranges".
- Fix LP:644813: SSL issues with DNS and embedded objects.
- Fix address space size check for SpreadAs.
- Fix multiple alias range support in the aka tool.
- Fix compilation of the msl_test tool on Linux.
- Make out-of-memory handler more robust.
- Minor fixes and code cleanup.
version 4.1.0
2010/05/13
- Support SOCKS5 proxies for HTTP, HTTPS, and passive FTP
traffic (socks_proxies and socks_prob PGL Robot fields). No
support for Active FTP data connections yet. "None" and basic
"username/password" SOCKS authentication methods are
supported. Chaining SOCKS and HTTP proxies is supported (the
socks_chaining_prob PGL Robot field).
- Extend the ipsToNames() PGL function to support a domain name
generation template with ${dashed_ip} and ${port} macros. This
change allows more flexibility in naming a large number of
server domains.
- Add a dynamize() PGL function that converts an array of static
domain names to dynamic names (using the supplied renewal
percentage), similar to what the dynamicName() function does
to one static name. This change allows creation of a large
number of servers, each with a unique dynamic domain name.
- Do not send "100 Continue" control response if we got the
entire request body already. This fix prevents POST requests
sent without waiting for 100 Continue response from getting
stuck because the client sent everything and cannot continue.
- Fix server freeze when "Expect: 100-continue" request headers
and body (partial or complete) is received in a single read(2)
call.
- Fix assertion when an HTTP client aborts transaction.
- Fix abort position calculation on the client side. The abort
size was set bigger than the request size, and client aborted
after the request was sent, during response reading.
version 4.0.11
2010/11/06
- Limit PGL load and population factors value to [0%, 100%] range.
Print error and exit if begin or end factor value is greater than
100%. If a script changes factor to a negative or greater than
100% value, print warning and set the factor to 0%/100% instead.
- Fixed PGL TblDistr to produce the same results for the same
seed. For example, if a table distribution is used for
response body sizes, the response size would stay the same
when the same URL is visited multiple times.
- Fixed several build and startup problems on Mac OS X.
- Fixed a DummyNet-related build problem on FreeBSD 8.1.
- Skip already configured network interface aliases when creating