原始日志numactl-N0-m0/home/test/spark-3.3.3-bin-hadoop3/bin/spark-sql \--master local[14]\--driver-java-options-Xloggc:/home/test/profiler/driver_gc.log -XX:PrintGCDetails -XX:PrintGCTimeStamps\--driver-memory90g \--database tpcds_parquet_tpcds_scale_200_db \-f/home/test/sql/q83.sql$ cat driver_gc.log OpenJDK64-Bit ServerVM(25.372-b07)forlinux-amd64JRE(1.8.0_372-b07),built on Apr19202308:57:57byjenkinswith gcc7.5.0Memory:4k page,physical788161268k(744451844k free),swap0k(0k free)CommandLine flags:-XX:IgnoreUnrecognizedVMOptions-XX:InitialHeapSize2147483648-XX:MaxHeapSize96636764160-XX:PrintGC-XX:PrintGCDetails-XX:PrintGCTimeStamps-XX:UseParallelGC4.014:[GC(Metadata GC Threshold)[PSYoungGen:188929K-16120K(611840K)]188929K-16144K(2010112K),0.0119549secs][Times:user0.04sys0.01,real0.01secs]4.026:[FullGC(Metadata GC Threshold)[PSYoungGen:16120K-0K(611840K)][ParOldGen:24K-15249K(540160K)]16144K-15249K(1152000K),[Metaspace:20356K-20356K(22528K)],0.0197266secs][Times:user0.12sys0.01,real0.02secs]6.637:[GC(Metadata GC Threshold)[PSYoungGen:141761K-16338K(611840K)]157011K-31660K(1152000K),0.0111804secs][Times:user0.03sys0.00,real0.00secs]6.649:[FullGC(Metadata GC Threshold)[PSYoungGen:16338K-0K(611840K)][ParOldGen:15321K-21851K(899072K)]31660K-21851K(1510912K),[Metaspace:33903K-33903K(36864K)],0.0263242secs][Times:user0.09sys0.03,real0.03secs]8.640:[GC(Metadata GC Threshold)[PSYoungGen:314953K-27022K(611840K)]336804K-48881K(1510912K),0.0138599secs][Times:user0.04sys0.02,real0.01secs]8.654:[FullGC(Metadata GC Threshold)[PSYoungGen:27022K-0K(611840K)][ParOldGen:21859K-36404K(1327104K)]48881K-36404K(1938944K),[Metaspace:56265K-56255K(59392K)],0.0656241secs][Times:user0.41sys0.00,real0.07secs]12.778:[GC(Allocation Failure)[PSYoungGen:524800K-45799K(691200K)]561204K-82212K(2018304K),0.0232692secs][Times:user0.06sys0.02,real0.02secs]13.651:[GC(Metadata GC Threshold)[PSYoungGen:177058K-26954K(762880K)]213471K-63391K(2089984K),0.0127219secs][Times:user0.09sys0.02,real0.01secs]13.664:[FullGC(Metadata GC Threshold)[PSYoungGen:26954K-0K(762880K)][ParOldGen:36436K-47198K(1857024K)]63391K-47198K(2619904K),[Metaspace:94603K-94603K(100352K)],0.0567371secs][Times:user0.33sys0.02,real0.06secs]19.404:[GC(Allocation Failure)[PSYoungGen:675840K-42995K(871424K)]723038K-112450K(2728448K),0.0311273secs][Times:user0.13sys0.02,real0.03secs]19.952:[GC(Allocation Failure)[PSYoungGen:871411K-51701K(890880K)]940866K-121172K(2747904K),0.0296181secs][Times:user0.18sys0.01,real0.03secs]20.218:[GC(Allocation Failure)[PSYoungGen:880117K-67582K(1347584K)]949588K-144965K(3204608K),0.1164347secs][Times:user0.78sys0.03,real0.11secs]20.679:[GC(Allocation Failure)[PSYoungGen:1347582K-81398K(1361408K)]1424965K-174117K(3218432K),0.1862208secs][Times:user1.04sys0.05,real0.18secs]21.187:[GC(GCLocker Initiated GC)[PSYoungGen:1361398K-75568K(2079232K)]1454117K-199370K(3936256K),0.1962536secs][Times:user1.12sys0.04,real0.20secs]21.931:[GC(Allocation Failure)[PSYoungGen:2054960K-90928K(2084864K)]2178767K-221756K(3941888K),0.2904077secs][Times:user1.58sys0.04,real0.29secs]22.735:[GC(GCLocker Initiated GC)[PSYoungGen:2070320K-115184K(3198464K)]2201162K-252834K(5055488K),0.2870335secs][Times:user1.62sys0.05,real0.29secs]24.994:[GC(Allocation Failure)[PSYoungGen:3198448K-134140K(3217408K)]3336103K-292387K(5074432K),0.4149455secs][Times:user2.31sys0.05,real0.42secs]26.166:[GC(Allocation Failure)[PSYoungGen:3217404K-139280K(4978688K)]3375651K-342576K(6835712K),0.4512913secs][Times:user2.49sys0.06,real0.46secs]27.426:[GC(Metadata GC Threshold)[PSYoungGen:2227994K-100740K(4995072K)]2431289K-314205K(6852096K),0.2786383secs][Times:user1.50sys0.04,real0.28secs]27.704:[FullGC(Metadata GC Threshold)[PSYoungGen:100740K-0K(4995072K)][ParOldGen:213464K-191716K(2071552K)]314205K-191716K(7066624K),[Metaspace:145232K-137818K(165888K)],0.1685947secs][Times:user1.01sys0.01,real0.17secs]29.746:[GC(GCLocker Initiated GC)[PSYoungGen:4816896K-177147K(7703040K)]5008616K-398346K(9774592K),0.6594221secs][Times:user3.70sys0.08,real0.66secs]35.203:[GC(Allocation Failure)[PSYoungGen:7703035K-214527K(7740416K)]7924269K-1964935K(9811968K),0.3887560secs][Times:user3.17sys0.89,real0.39secs]35.592:[FullGC(Ergonomics)[PSYoungGen:214527K-0K(7740416K)][ParOldGen:1750407K-1180998K(3698176K)]1964935K-1180998K(11438592K),[Metaspace:168023K-164336K(188416K)],0.4731631secs][Times:user2.58sys0.07,real0.47secs]Heap PSYoungGen total7740416K,used2221660K[0x00001503dd000000,0x00001506b8900000,0x0000150b5d000000)eden space7525888K,29%used[0x00001503dd000000,0x0000150464997320,0x00001505a8580000)from space214528K,0%used[0x00001505a8580000,0x00001505a8580000,0x00001505b5700000)to space813568K,0%used[0x0000150686e80000,0x0000150686e80000,0x00001506b8900000)ParOldGen total3698176K,used1180998K[0x000014f4dd000000,0x000014f5beb80000,0x00001503dd000000)object space3698176K,31%used[0x000014f4dd000000,0x000014f525151870,0x000014f5beb80000)Metaspace used166650K,capacity181271K,committed188160K,reserved188416KMetadata GC Threshold只要 GC 原因是 Metadata GC ThresholdMinor GC 之后必跟一次 Full GC。