docker ビルド時のレイヤー上限は 125

docker ビルド時のレイヤー上限は 125 (暫定) という話。終わり。

動作確認

OS は Ubuntu 、docker は 20.10.2 を利用する。

# docker --version
Docker version 20.10.2, build 20.10.2-0ubuntu1~20.04.2

以下のようなファイルを用意してイメージをビルドする。

FROM alpine:latest
WORKDIR /app
RUN touch 1.txt
RUN touch 2.txt
RUN touch 3.txt
...

すると、Step 126 で max depth exceeded でエラーになる。

# docker build -t too-many-layers -f Dockerfile .
Sending build context to Docker daemon  6.144kB
Step 1/127 : FROM alpine:latest
 ---> 6dbb9cc54074
Step 2/127 : WORKDIR /app
 ---> Running in 035bdb197382
Removing intermediate container 035bdb197382
 ---> 14ab0955f1a6
Step 3/127 : RUN touch 1.txt
 ---> Running in db3ece39cf58

...

Step 126/127 : RUN touch 124.txt
 ---> Running in a192710bcc70
max depth exceeded

なんで?

以下のような Issue があった。

github.com

実装上は次のコードでエラーを出していた。 moby/moby layer/layer_store.go

       if p.depth() >= maxLayerDepth {
            err = ErrMaxDepthExceeded
            return nil, err
        }

コメントを見ると、graphdrivers の上限が 127 なので rwlayer のために 2 減らして 125 としているらしい。

// maxLayerDepth represents the maximum number of
// layers which can be chained together. 125 was
// chosen to account for the 127 max in some
// graphdrivers plus the 2 additional layers
// used to create a rwlayer.
const maxLayerDepth = 125

いくつかの graphdriver ではコンテナ実行時にもレイヤー数をチェックしており、これも同様の上限になっていた。 moby/moby daemon/graphdriver/overlay2/overlay.go

こちらの記載のほうが分かりやすいが、 mount コマンドの上限による都合の様子。

   maxDepth      = 128

    // idLength represents the number of random characters
    // which can be used to create the unique link identifier
    // for every layer. If this value is too long then the
    // page size limit for the mount command may be exceeded.
    // The idLength should be selected such that following equation
    // is true (512 is a buffer for label metadata).
    // ((idLength + len(linkDir) + 1) * maxDepth) <= (pageSize - 512)

試しに overlayfs でたくさんのレイヤーを重ねてみたが、ある程度の長さで mount コマンドが失敗している。

# mkdir layers merged overlay upper work
# seq 1 1000 | xargs -I @@ bash -c "mkdir layers/@@ ; touch layers/@@/@@.txt"

# mount -t overlay overlay -o lowerdir=$(seq 1 100 | sed "s#^#layers/#g" | tr "\n" ":" | sed "s/:$//g"),upperdir=upper,workdir=work merged
# ls merged/
100.txt  13.txt  17.txt  20.txt  24.txt  28.txt  31.txt  35.txt  39.txt  42.txt  46.txt  4.txt   53.txt  57.txt  60.txt  64.txt  68.txt  71.txt  75.txt  79.txt  82.txt  86.txt  8.txt   93.txt  97.txt
10.txt   14.txt  18.txt  21.txt  25.txt  29.txt  32.txt  36.txt  3.txt   43.txt  47.txt  50.txt  54.txt  58.txt  61.txt  65.txt  69.txt  72.txt  76.txt  7.txt   83.txt  87.txt  90.txt  94.txt  98.txt
11.txt   15.txt  19.txt  22.txt  26.txt  2.txt   33.txt  37.txt  40.txt  44.txt  48.txt  51.txt  55.txt  59.txt  62.txt  66.txt  6.txt   73.txt  77.txt  80.txt  84.txt  88.txt  91.txt  95.txt  99.txt
12.txt   16.txt  1.txt   23.txt  27.txt  30.txt  34.txt  38.txt  41.txt  45.txt  49.txt  52.txt  56.txt  5.txt   63.txt  67.txt  70.txt  74.txt  78.txt  81.txt  85.txt  89.txt  92.txt  96.txt  9.txt
# umount merged

# mount -t overlay overlay -o lowerdir=$(seq 1 1000 | sed "s#^#layers/#g" | tr "\n" ":" | sed "s/:$//g"),upperdir=upper,workdir=work merged
mount: /tmp/fs/merged: mount(2) system call failed: No such file or directory.

mount コマンドに渡すまえに bash でエラーになってなりしないか確認したが、strace コマンドの結果から引数は正常そうなことがわかる。そのため、mount システムコール内で失敗している思われる。

# strace -s 1000000 -e mount mount -t overlay overlay -o lowerdir=$(seq 1 1000 | sed "s#^#layers/#g" | tr "\n" ":" | sed "s/:$//g"),upperdir=upper,workdir=work merged
mount("overlay", "/tmp/fs/merged", "overlay", 0, "lowerdir=layers/1:layers/2:layers/3:layers/4:layers/5:layers/6:layers/7:layers/8:layers/9:layers/10:layers/11:layers/12:layers/13:layers/14:layers/15:layers/16:layers/17:layers/18:layers/19:layers/20:layers/21:layers/22:layers/23:layers/24:layers/25:layers/26:layers/27:layers/28:layers/29:layers/30:layers/31:layers/32:layers/33:layers/34:layers/35:layers/36:layers/37:layers/38:layers/39:layers/40:layers/41:layers/42:layers/43:layers/44:layers/45:layers/46:layers/47:layers/48:layers/49:layers/50:layers/51:layers/52:layers/53:layers/54:layers/55:layers/56:layers/57:layers/58:layers/59:layers/60:layers/61:layers/62:layers/63:layers/64:layers/65:layers/66:layers/67:layers/68:layers/69:layers/70:layers/71:layers/72:layers/73:layers/74:layers/75:layers/76:layers/77:layers/78:layers/79:layers/80:layers/81:layers/82:layers/83:layers/84:layers/85:layers/86:layers/87:layers/88:layers/89:layers/90:layers/91:layers/92:layers/93:layers/94:layers/95:layers/96:layers/97:layers/98:layers/99:layers/100:layers/101:layers/102:layers/103:layers/104:layers/105:layers/106:layers/107:layers/108:layers/109:layers/110:layers/111:layers/112:layers/113:layers/114:layers/115:layers/116:layers/117:layers/118:layers/119:layers/120:layers/121:layers/122:layers/123:layers/124:layers/125:layers/126:layers/127:layers/128:layers/129:layers/130:layers/131:layers/132:layers/133:layers/134:layers/135:layers/136:layers/137:layers/138:layers/139:layers/140:layers/141:layers/142:layers/143:layers/144:layers/145:layers/146:layers/147:layers/148:layers/149:layers/150:layers/151:layers/152:layers/153:layers/154:layers/155:layers/156:layers/157:layers/158:layers/159:layers/160:layers/161:layers/162:layers/163:layers/164:layers/165:layers/166:layers/167:layers/168:layers/169:layers/170:layers/171:layers/172:layers/173:layers/174:layers/175:layers/176:layers/177:layers/178:layers/179:layers/180:layers/181:layers/182:layers/183:layers/184:layers/185:layers/186:layers/187:layers/188:layers/189:layers/190:layers/191:layers/192:layers/193:layers/194:layers/195:layers/196:layers/197:layers/198:layers/199:layers/200:layers/201:layers/202:layers/203:layers/204:layers/205:layers/206:layers/207:layers/208:layers/209:layers/210:layers/211:layers/212:layers/213:layers/214:layers/215:layers/216:layers/217:layers/218:layers/219:layers/220:layers/221:layers/222:layers/223:layers/224:layers/225:layers/226:layers/227:layers/228:layers/229:layers/230:layers/231:layers/232:layers/233:layers/234:layers/235:layers/236:layers/237:layers/238:layers/239:layers/240:layers/241:layers/242:layers/243:layers/244:layers/245:layers/246:layers/247:layers/248:layers/249:layers/250:layers/251:layers/252:layers/253:layers/254:layers/255:layers/256:layers/257:layers/258:layers/259:layers/260:layers/261:layers/262:layers/263:layers/264:layers/265:layers/266:layers/267:layers/268:layers/269:layers/270:layers/271:layers/272:layers/273:layers/274:layers/275:layers/276:layers/277:layers/278:layers/279:layers/280:layers/281:layers/282:layers/283:layers/284:layers/285:layers/286:layers/287:layers/288:layers/289:layers/290:layers/291:layers/292:layers/293:layers/294:layers/295:layers/296:layers/297:layers/298:layers/299:layers/300:layers/301:layers/302:layers/303:layers/304:layers/305:layers/306:layers/307:layers/308:layers/309:layers/310:layers/311:layers/312:layers/313:layers/314:layers/315:layers/316:layers/317:layers/318:layers/319:layers/320:layers/321:layers/322:layers/323:layers/324:layers/325:layers/326:layers/327:layers/328:layers/329:layers/330:layers/331:layers/332:layers/333:layers/334:layers/335:layers/336:layers/337:layers/338:layers/339:layers/340:layers/341:layers/342:layers/343:layers/344:layers/345:layers/346:layers/347:layers/348:layers/349:layers/350:layers/351:layers/352:layers/353:layers/354:layers/355:layers/356:layers/357:layers/358:layers/359:layers/360:layers/361:layers/362:layers/363:layers/364:layers/365:layers/366:layers/367:layers/368:layers/369:layers/370:layers/371:layers/372:layers/373:layers/374:layers/375:layers/376:layers/377:layers/378:layers/379:layers/380:layers/381:layers/382:layers/383:layers/384:layers/385:layers/386:layers/387:layers/388:layers/389:layers/390:layers/391:layers/392:layers/393:layers/394:layers/395:layers/396:layers/397:layers/398:layers/399:layers/400:layers/401:layers/402:layers/403:layers/404:layers/405:layers/406:layers/407:layers/408:layers/409:layers/410:layers/411:layers/412:layers/413:layers/414:layers/415:layers/416:layers/417:layers/418:layers/419:layers/420:layers/421:layers/422:layers/423:layers/424:layers/425:layers/426:layers/427:layers/428:layers/429:layers/430:layers/431:layers/432:layers/433:layers/434:layers/435:layers/436:layers/437:layers/438:layers/439:layers/440:layers/441:layers/442:layers/443:layers/444:layers/445:layers/446:layers/447:layers/448:layers/449:layers/450:layers/451:layers/452:layers/453:layers/454:layers/455:layers/456:layers/457:layers/458:layers/459:layers/460:layers/461:layers/462:layers/463:layers/464:layers/465:layers/466:layers/467:layers/468:layers/469:layers/470:layers/471:layers/472:layers/473:layers/474:layers/475:layers/476:layers/477:layers/478:layers/479:layers/480:layers/481:layers/482:layers/483:layers/484:layers/485:layers/486:layers/487:layers/488:layers/489:layers/490:layers/491:layers/492:layers/493:layers/494:layers/495:layers/496:layers/497:layers/498:layers/499:layers/500:layers/501:layers/502:layers/503:layers/504:layers/505:layers/506:layers/507:layers/508:layers/509:layers/510:layers/511:layers/512:layers/513:layers/514:layers/515:layers/516:layers/517:layers/518:layers/519:layers/520:layers/521:layers/522:layers/523:layers/524:layers/525:layers/526:layers/527:layers/528:layers/529:layers/530:layers/531:layers/532:layers/533:layers/534:layers/535:layers/536:layers/537:layers/538:layers/539:layers/540:layers/541:layers/542:layers/543:layers/544:layers/545:layers/546:layers/547:layers/548:layers/549:layers/550:layers/551:layers/552:layers/553:layers/554:layers/555:layers/556:layers/557:layers/558:layers/559:layers/560:layers/561:layers/562:layers/563:layers/564:layers/565:layers/566:layers/567:layers/568:layers/569:layers/570:layers/571:layers/572:layers/573:layers/574:layers/575:layers/576:layers/577:layers/578:layers/579:layers/580:layers/581:layers/582:layers/583:layers/584:layers/585:layers/586:layers/587:layers/588:layers/589:layers/590:layers/591:layers/592:layers/593:layers/594:layers/595:layers/596:layers/597:layers/598:layers/599:layers/600:layers/601:layers/602:layers/603:layers/604:layers/605:layers/606:layers/607:layers/608:layers/609:layers/610:layers/611:layers/612:layers/613:layers/614:layers/615:layers/616:layers/617:layers/618:layers/619:layers/620:layers/621:layers/622:layers/623:layers/624:layers/625:layers/626:layers/627:layers/628:layers/629:layers/630:layers/631:layers/632:layers/633:layers/634:layers/635:layers/636:layers/637:layers/638:layers/639:layers/640:layers/641:layers/642:layers/643:layers/644:layers/645:layers/646:layers/647:layers/648:layers/649:layers/650:layers/651:layers/652:layers/653:layers/654:layers/655:layers/656:layers/657:layers/658:layers/659:layers/660:layers/661:layers/662:layers/663:layers/664:layers/665:layers/666:layers/667:layers/668:layers/669:layers/670:layers/671:layers/672:layers/673:layers/674:layers/675:layers/676:layers/677:layers/678:layers/679:layers/680:layers/681:layers/682:layers/683:layers/684:layers/685:layers/686:layers/687:layers/688:layers/689:layers/690:layers/691:layers/692:layers/693:layers/694:layers/695:layers/696:layers/697:layers/698:layers/699:layers/700:layers/701:layers/702:layers/703:layers/704:layers/705:layers/706:layers/707:layers/708:layers/709:layers/710:layers/711:layers/712:layers/713:layers/714:layers/715:layers/716:layers/717:layers/718:layers/719:layers/720:layers/721:layers/722:layers/723:layers/724:layers/725:layers/726:layers/727:layers/728:layers/729:layers/730:layers/731:layers/732:layers/733:layers/734:layers/735:layers/736:layers/737:layers/738:layers/739:layers/740:layers/741:layers/742:layers/743:layers/744:layers/745:layers/746:layers/747:layers/748:layers/749:layers/750:layers/751:layers/752:layers/753:layers/754:layers/755:layers/756:layers/757:layers/758:layers/759:layers/760:layers/761:layers/762:layers/763:layers/764:layers/765:layers/766:layers/767:layers/768:layers/769:layers/770:layers/771:layers/772:layers/773:layers/774:layers/775:layers/776:layers/777:layers/778:layers/779:layers/780:layers/781:layers/782:layers/783:layers/784:layers/785:layers/786:layers/787:layers/788:layers/789:layers/790:layers/791:layers/792:layers/793:layers/794:layers/795:layers/796:layers/797:layers/798:layers/799:layers/800:layers/801:layers/802:layers/803:layers/804:layers/805:layers/806:layers/807:layers/808:layers/809:layers/810:layers/811:layers/812:layers/813:layers/814:layers/815:layers/816:layers/817:layers/818:layers/819:layers/820:layers/821:layers/822:layers/823:layers/824:layers/825:layers/826:layers/827:layers/828:layers/829:layers/830:layers/831:layers/832:layers/833:layers/834:layers/835:layers/836:layers/837:layers/838:layers/839:layers/840:layers/841:layers/842:layers/843:layers/844:layers/845:layers/846:layers/847:layers/848:layers/849:layers/850:layers/851:layers/852:layers/853:layers/854:layers/855:layers/856:layers/857:layers/858:layers/859:layers/860:layers/861:layers/862:layers/863:layers/864:layers/865:layers/866:layers/867:layers/868:layers/869:layers/870:layers/871:layers/872:layers/873:layers/874:layers/875:layers/876:layers/877:layers/878:layers/879:layers/880:layers/881:layers/882:layers/883:layers/884:layers/885:layers/886:layers/887:layers/888:layers/889:layers/890:layers/891:layers/892:layers/893:layers/894:layers/895:layers/896:layers/897:layers/898:layers/899:layers/900:layers/901:layers/902:layers/903:layers/904:layers/905:layers/906:layers/907:layers/908:layers/909:layers/910:layers/911:layers/912:layers/913:layers/914:layers/915:layers/916:layers/917:layers/918:layers/919:layers/920:layers/921:layers/922:layers/923:layers/924:layers/925:layers/926:layers/927:layers/928:layers/929:layers/930:layers/931:layers/932:layers/933:layers/934:layers/935:layers/936:layers/937:layers/938:layers/939:layers/940:layers/941:layers/942:layers/943:layers/944:layers/945:layers/946:layers/947:layers/948:layers/949:layers/950:layers/951:layers/952:layers/953:layers/954:layers/955:layers/956:layers/957:layers/958:layers/959:layers/960:layers/961:layers/962:layers/963:layers/964:layers/965:layers/966:layers/967:layers/968:layers/969:layers/970:layers/971:layers/972:layers/973:layers/974:layers/975:layers/976:layers/977:layers/978:layers/979:layers/980:layers/981:layers/982:layers/983:layers/984:layers/985:layers/986:layers/987:layers/988:layers/989:layers/990:layers/991:layers/992:layers/993:layers/994:layers/995:layers/996:layers/997:layers/998:layers/999:layers/1000,upperdir=upper,workdir=work") = -1 ENOENT (No such file or directory)

dmesg には次のようなエラーが出ていた。

# dmesg | tail
...
[73448.037990] overlayfs: failed to resolve 'lay': -2

カーネルコードはおおむね次のようになっており、1ページ分(4k)の領域しか確保していないことがわかる(apt で src 取ってきてないので微妙に違うかもしれないけどまあ大まかには一緒でしょうたぶん)。 fs/namespace.c#copy_mount_options

void *copy_mount_options(const void __user * data)
{
    int i;
    unsigned long size;
    char *copy;

    if (!data)
        return NULL;

    copy = kmalloc(PAGE_SIZE, GFP_KERNEL);
    if (!copy)
        return ERR_PTR(-ENOMEM);

    /* We only care that *some* data at the address the user
    * gave us is valid.  Just in case, we'll zero
    * the remainder of the page.
    */
    /* copy_from_user cannot cross TASK_SIZE ! */
    size = TASK_SIZE - (unsigned long)untagged_addr(data);
    if (size > PAGE_SIZE)
        size = PAGE_SIZE;

    i = size - exact_copy_from_user(copy, data, size);
    if (!i) {
        kfree(copy);
        return ERR_PTR(-EFAULT);
    }
    if (i != PAGE_SIZE)
        memset(copy + i, 0, PAGE_SIZE - i);
    return copy;
}

まとめ

ということで、docker ビルド時のレイヤー上限は 125。 ただし docker のバージョンによって変わってきた数字(昔は aufs が制限厳しかったので 42 だったらしい)だし、今後も実装依存で変わる可能性もあるので、ドキュメント化はされないと思われる。