Merge trunk

chapter05/glibc.xml

@@ -94,7 +94,6 @@ cd       build</userinput></screen>
       --host=$LFS_TGT                    \
       --build=$(../scripts/config.guess) \
       --enable-kernel=&min-kernel;               \
-      --with-headers=$LFS/usr/include    \
       --disable-nscd                     \
       libc_cv_slibdir=/usr/lib</userinput></screen>
 
@@ -119,16 +118,6 @@ cd       build</userinput></screen>
         </listitem>
       </varlistentry>
 
-      <varlistentry>
-        <term><parameter>--with-headers=$LFS/usr/include</parameter></term>
-        <listitem>
-          <para>This tells Glibc to compile itself against the headers
-          recently installed to the $LFS/usr/include directory, so that
-          it knows exactly what features the kernel has and can optimize
-          itself accordingly.</para>
-        </listitem>
-      </varlistentry>
-
       <varlistentry>
         <term><parameter>libc_cv_slibdir=/usr/lib</parameter></term>
         <listitem>
@@ -192,7 +181,7 @@ cd       build</userinput></screen>
           class="directory">/</filename>) directory. Here we specify that
           the package is installed in <filename class="directory">
           $LFS</filename>, which will become the root directory in <xref linkend=
-          "ch-tools-chroot"/>.</para>
+          "ch-tools-chroot" role='.'/></para>
         </listitem>
       </varlistentry>
 
@@ -203,32 +192,101 @@ cd       build</userinput></screen>
 
 <screen><userinput remap="install">sed '/RTLDLIST=/s@/usr@@g' -i $LFS/usr/bin/ldd</userinput></screen>
 
-    <caution>
+    <para>Now that our cross toolchain is in place, it is important to ensure
-      <para>At this point, it is imperative to stop and ensure that the basic
+    that compiling and linking will work as expected. We do this by performing
-      functions (compiling and linking) of the new toolchain are working as
+    some sanity checks:</para>
-      expected. To perform a sanity check, run the following commands:</para>
 
-<screen><userinput>echo 'int main(){}' | $LFS_TGT-gcc -xc -
+<screen><userinput>echo 'int main(){}' | $LFS_TGT-gcc -x c - -v -Wl,--verbose &amp;&gt; dummy.log
-readelf -l a.out | grep ld-linux</userinput></screen>
+readelf -l a.out | grep ': /lib'</userinput></screen>
 
-      <para>If everything is working correctly, there should be no errors,
+  <para>There should be no errors,
-      and the output of the last command will be of the form:</para>
+  and the output of the last command will be (allowing for
+  platform-specific differences in the dynamic linker name):</para>
 
 <screen><computeroutput>[Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]</computeroutput></screen>
 
-      <para arch="default">Note that for 32-bit machines, the interpreter name will be
+  <para arch="default">Note that for 32-bit machines, the interpreter name will be
-      <filename>/lib/ld-linux.so.2</filename>.</para>
+  <filename>/lib/ld-linux.so.2</filename>.</para>
 
-      <para>If the output is not as shown above, or there is no output at all,
+  <para>Note that this path should not contain
-      then something is wrong. Investigate and retrace the steps to find out
+  <filename class='directory'>/mnt/lfs</filename> (or the value of
-      where the problem is and correct it. This issue must be resolved before
+  the <envar>LFS</envar> variable if you used a different one).  The path is
-      continuing.</para>
+  resolved when the compiled program is executed, and that should only happen
+  after we enter the chroot environment where the kernel would consider
+  <filename class='directory'>$LFS</filename> as the root directory
+  (<filename class='directory'>/</filename>).</para>
 
-      <para>Once all is well, clean up the test file:</para>
+  <para>Now make sure that we're set up to use the correct start files:</para>
 
-<screen><userinput>rm -v a.out</userinput></screen>
+<screen><userinput>grep -E -o "$LFS/lib.*/S?crt[1in].*succeeded" dummy.log</userinput></screen>
 
-    </caution>
+  <para>The output of the last command should be:</para>
+
+<screen><computeroutput>/mnt/lfs/lib/../lib/Scrt1.o succeeded
+/mnt/lfs/lib/../lib/crti.o succeeded
+/mnt/lfs/lib/../lib/crtn.o succeeded</computeroutput></screen>
+
+  <para>Verify that the compiler is searching for the correct header
+  files:</para>
+
+<screen><userinput>grep -B3 "^ $LFS/usr/include" dummy.log</userinput></screen>
+
+  <para>This command should return the following output:</para>
+
+<screen><computeroutput>#include &lt;...&gt; search starts here:
+ /mnt/lfs/tools/lib/gcc/x86_64-lfs-linux-gnu/&gcc-version;/include
+ /mnt/lfs/tools/lib/gcc/x86_64-lfs-linux-gnu/&gcc-version;/include-fixed
+ /mnt/lfs/usr/include</computeroutput></screen>
+
+   <para>Again, the directory named after your target triplet may be
+   different than the above, depending on your system architecture.</para>
+
+  <para>Next, verify that the new linker is being used with the correct search paths:</para>
+
+<screen><userinput>grep 'SEARCH.*/usr/lib' dummy.log |sed 's|; |\n|g'</userinput></screen>
+
+  <para>References to paths that have components with '-linux-gnu' should
+  be ignored, but otherwise the output of the last command should be:</para>
+
+<screen><computeroutput>SEARCH_DIR("=/mnt/lfs/tools/x86_64-lfs-linux-gnu/lib64")
+SEARCH_DIR("=/usr/local/lib64")
+SEARCH_DIR("=/lib64")
+SEARCH_DIR("=/usr/lib64")
+SEARCH_DIR("=/mnt/lfs/tools/x86_64-lfs-linux-gnu/lib")
+SEARCH_DIR("=/usr/local/lib")
+SEARCH_DIR("=/lib")
+SEARCH_DIR("=/usr/lib");</computeroutput></screen>
+
+   <para>A 32-bit system may use a few other directories, but anyway
+   the important facet here is all the paths should begin with an equal sign
+   (<literal>=</literal>), which would be replaced with the sysroot
+   directory that we've configured for the linker.</para>
+
+  <para>Next make sure that we're using the correct libc:</para>
+
+<screen><userinput>grep "/lib.*/libc.so.6 " dummy.log</userinput></screen>
+
+  <para>The output of the last command should be:</para>
+
+<screen><computeroutput>attempt to open /mnt/lfs/usr/lib/libc.so.6 succeeded</computeroutput></screen>
+
+  <para>Make sure GCC is using the correct dynamic linker:</para>
+
+<screen><userinput>grep found dummy.log</userinput></screen>
+
+  <para>The output of the last command should be (allowing for
+  platform-specific differences in dynamic linker name):</para>
+
+<screen><computeroutput>found ld-linux-x86-64.so.2 at /mnt/lfs/usr/lib/ld-linux-x86-64.so.2</computeroutput></screen>
+
+  <para>If the output does not appear as shown above or is not received
+  at all, then something is seriously wrong. Investigate and retrace the
+  steps to find out where the problem is and correct it.  Any
+  issues should be resolved before continuing with the process.</para>
+
+  <para>Once everything is working correctly, clean up the test files:</para>
+
+<screen><userinput>rm -v a.out dummy.log</userinput></screen>
 
     <note><para>Building the packages in the next chapter will serve as an
     additional check that the toolchain has been built properly. If some

chapter08/gcc.xml

@@ -251,8 +251,7 @@ su tester -c "PATH=$PATH make -k check"</userinput></screen>
     that compiling and linking will work as expected. We do this by performing
     some sanity checks:</para>
 
-<screen><userinput>echo 'int main(){}' &gt; dummy.c
+<screen><userinput>echo 'int main(){}' | cc dummy.c -x c - -v -Wl,--verbose &amp;&gt; dummy.log
-cc dummy.c -v -Wl,--verbose &amp;&gt; dummy.log
 readelf -l a.out | grep ': /lib'</userinput></screen>
 
   <para>There should be no errors,
@@ -348,7 +347,7 @@ SEARCH_DIR("/usr/lib");</computeroutput></screen>
 
   <para>Once everything is working correctly, clean up the test files:</para>
 
-<screen><userinput>rm -v dummy.c a.out dummy.log</userinput></screen>
+<screen><userinput>rm -v a.out dummy.log</userinput></screen>
 
     <para>Finally, move a misplaced file:</para>
 

part3intro/toolchaintechnotes.xml

@@ -3,6 +3,9 @@
   "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [
   <!ENTITY % general-entities SYSTEM "../general.ent">
   %general-entities;
+
+  <!ENTITY host-triplet
+           "<replaceable>&lt;the host triplet&gt;</replaceable>">
 ]>
 
 <sect1 id="ch-tools-toolchaintechnotes" xreflabel="Toolchain Technical Notes">
@@ -44,6 +47,14 @@
         book for a cross-toolchain for some purpose other
         than building LFS, unless you really understand what you are doing.
       </para>
+
+      <para>
+        It's known installing GCC pass 2 will break the cross-toolchain.
+        We don't consider it a bug because GCC pass 2 is the last package
+        to be cross-compiled in the book, and we won't <quote>fix</quote>
+        it until we really need to cross-compile some package after GCC
+        pass 2 in the future.
+      </para>
     </note>
 
     <para>Cross-compilation involves some concepts that deserve a section of
@@ -197,14 +208,105 @@
       page</ulink>.</para>
     </note>
 
-    <para>In order to fake a cross-compilation in LFS, the name of the host triplet
+    <para>
-    is slightly adjusted by changing the &quot;vendor&quot; field in the
+      There are two key points for a cross-compilation:
-    <envar>LFS_TGT</envar> variable so it says &quot;lfs&quot;. We also use the
+    </para>
-    <parameter>--with-sysroot</parameter> option when building the cross-linker and
+
-    cross-compiler, to tell them where to find the needed host files. This
+    <itemizedlist>
-    ensures that none of the other programs built in <xref
+      <listitem>
-    linkend="chapter-temporary-tools"/> can link to libraries on the build
+        <para>
-    machine. Only two stages are mandatory, plus one more for tests.</para>
+          When producing and processing the machine code supposed to be
+          executed on <quote>the host,</quote> the cross-toolchain must be
+          used.  Note that the native toolchain from <quote>the build</quote>
+          may be still invoked to generate machine code supposed to be
+          executed on <quote>the build.</quote>  For example, the build system
+          may compile a generator with the native toolchain, then generate
+          a C source file with the generator, and finally compile the C
+          source file with the cross-toolchain so the generated code will
+          be able to run on <quote>the host.</quote>
+        </para>
+        <para>
+          With an autoconf-based build system, this requirement is ensured
+          by using the <parameter>--host</parameter> switch to specify
+          <quote>the host</quote> triplet.  With this switch the build
+          system will use the toolchain components prefixed
+          with <literal>&host-triplet;</literal>
+          for generating and processing the machine code for
+          <quote>the host</quote>; e.g. the compiler will be
+          <command>&host-triplet;-gcc</command> and the
+          <command>readelf</command> tool will be
+          <command>&host-triplet;-readelf</command>.
+        </para>
+      </listitem>
+      <listitem>
+        <para>
+          The build system should not attempt to run any generated machine
+          code supposed to be executed on <quote>the host.</quote>  For
+          example, when building an utility natively, its man page can be
+          generated by running the utility with the
+          <parameter>--help</parameter> switch and processing the output,
+          but generally it's not possible to do so for a cross-compilation
+          as the utility may fail
+          to run on <quote>the build</quote>: it's obviously impossible to
+          run ARM64 machine code on a x86 CPU (without an emulator).
+        </para>
+        <para>
+          With an autoconf-based build system, this requirement is
+          satisfied in <quote>the cross-compilation mode</quote> where
+          the optional features requiring to run machine code for
+          <quote>the host</quote> during the build time are disabled.  When <quote>the
+          host</quote> triplet is explicitly specified, <quote>the
+          cross-compilation mode</quote> is enabled if and only if either
+          the <command>configure</command> script fails to run a dummy
+          program compiled into <quote>the host</quote> machine code, or
+          <quote>the build</quote> triplet is explicitly specified via the
+          <parameter>--build</parameter> switch and it's different from
+          <quote>the host</quote> triplet.
+        </para>
+      </listitem>
+    </itemizedlist>
+
+    <para>In order to cross-compile a package for the LFS temporary system,
+    the name of the system triplet is slightly adjusted by changing the
+    &quot;vendor&quot; field in the <envar>LFS_TGT</envar> variable so it
+    says &quot;lfs&quot; and <envar>LFS_TGT</envar> is then specified as
+    <quote>the host</quote> triplet via <parameter>--host</parameter>, so
+    the cross-toolchain will be used for generating and processing the
+    machine code running as a part of the LFS temporary system.  And, we
+    also need to enable <quote>the cross-compilation mode</quote>: despite
+    <quote>the host</quote> machine code, i.e. the machine code for the LFS
+    temporary system, is able to execute on the current CPU, it may refer
+    to a library not available on the <quote>the build</quote> (the host
+    distro), or some code or data non-exist or defined differently in the
+    library even if it happens to be available.  When cross-compiling a
+    package for the LFS temporary system, we cannot rely on the
+    <command>configure</command> script to detect this issue with the
+    dummy program: the dummy only uses a few components in
+    <systemitem class='library'>libc</systemitem> that the host distro
+    <systemitem class='library'>libc</systemitem> likely provides (unless,
+    maybe the host distro uses a different
+    <systemitem class='library'>libc</systemitem> implementaion like Musl),
+    so it won't fail like how the really useful programs would likely.
+    Thus we must explicitly specify <quote>the build</quote> triplet to
+    enable <quote>the cross-compilation mode.</quote>  The value we use is
+    just the default, i.e. the original system triplet from
+    <command>config.guess</command> output, but <quote>the cross-compilation
+    mode</quote> depends on an explicit specification as we've
+    discussed.</para>
+
+    <para>We use the <parameter>--with-sysroot</parameter> option when
+    building the cross-linker and cross-compiler, to tell them where to find
+    the needed files for <quote>the host.</quote> This nearly ensures that
+    none of the other programs built in
+    <xref linkend="chapter-temporary-tools"/> can link to libraries on
+    <quote>the build.</quote> The word <quote>nearly</quote> is used because
+    <command>libtool</command>, a <quote>compatibility</quote> wrapper of
+    the compiler and the linker for autoconf-based build systems,
+    can try to be too clever and mistakenly pass options allowing the linker
+    to find libraries of <quote>the build.</quote>
+    To prevent this fallout, we need to delete the libtool archive
+    (<filename class='extension'>.la</filename>) files and fix up an
+    outdated libtool copy shipped with the Binutils code.</para>
 
     <informaltable align="center">
       <tgroup cols="5">
@@ -228,7 +330,7 @@
           </row>
           <row>
             <entry>3</entry><entry>lfs</entry><entry>lfs</entry><entry>lfs</entry>
-            <entry>Rebuild and test cc-lfs using cc-lfs on lfs.</entry>
+            <entry>Rebuild (and maybe test) cc-lfs using cc-lfs on lfs.</entry>
           </row>
         </tbody>
       </tgroup>
@@ -256,30 +358,11 @@
     <para>The upshot of the preceding paragraph is that cc1 is unable to
     build a fully functional libstdc++ with the degraded libgcc, but cc1
     is the only compiler available for building the C/C++ libraries
-    during stage 2. There are two reasons we don't immediately use the
+    during stage 2.  As we've discussed, we cannot run cc-lfs on pc (the
-    compiler built in stage 2, cc-lfs, to build those libraries.</para>
+    host distro) because it may require some library, code, or data not
-
+    available on <quote>the build</quote> (the host distro).
-    <itemizedlist>
+    So when we build gcc stage 2, we override the library
-      <listitem>
+    search path to link libstdc++ against the newly
-        <para>
-          Generally speaking, cc-lfs cannot run on pc (the host system).  Even though the
-          triplets for pc and lfs are compatible with each other, an executable
-          for lfs must depend on glibc-&glibc-version;; the host distro
-          may utilize either a different implementation of libc (for example, musl), or
-          a previous release of glibc (for example, glibc-2.13).
-        </para>
-      </listitem>
-      <listitem>
-        <para>
-          Even if cc-lfs can run on pc, using it on pc would create
-          a risk of linking to the pc libraries, since cc-lfs is a native
-          compiler.
-        </para>
-      </listitem>
-    </itemizedlist>
-
-    <para>So when we build gcc stage 2, we instruct the building system to
-    rebuild libgcc and libstdc++ with cc1, but we link libstdc++ to the newly
     rebuilt libgcc instead of the old, degraded build.  This makes the rebuilt
     libstdc++ fully functional.</para>
 
@@ -290,12 +373,11 @@
     package on a completed LFS system, the reinstalled content of the package
     should be the same as the content of the same package when first installed in
     &ch-final;.  The temporary packages installed in &ch-tmp-cross; or
-    &ch-tmp-chroot; cannot satisfy this requirement, because some of them
+    &ch-tmp-chroot; cannot satisfy this requirement, because some optional
-    are built without optional dependencies, and autoconf cannot
+    features of them are disabled because of either the missing
-    perform some feature checks in &ch-tmp-cross; because of cross-compilation,
+    dependencies or the <quote>cross-compilation mode.</quote>
-    causing the temporary packages to lack optional features,
+    Additionally, a minor reason for rebuilding the packages is to run the
-    or use suboptimal code routines. Additionally, a minor reason for
+    test suites.</para>
-    rebuilding the packages is to run the test suites.</para>
 
   </sect2>
 
@@ -357,39 +439,25 @@ checking what linker to use... /mnt/lfs/tools/i686-lfs-linux-gnu/bin/ld</compute
     standard C library (glibc) to interface with features that the Linux
     kernel will provide.</para>
 
-    <para>Next comes glibc. The most important
+    <para>Next comes glibc. This is the first package that we cross-compile.
-    considerations for building glibc are the compiler, binary tools, and
+    We use the <parameter>--host=$LFS_TGT</parameter> option to make
-    kernel headers. The compiler and binary tools are generally not an issue
+    the build system to use those tools prefixed with
-    since glibc will always use those relating to the <parameter>--host</parameter>
+    <literal>$LFS_TGT-</literal>, and the
-    parameter passed to its configure script; e.g., in our case, the compiler
+    <parameter>--build=$(../scripts/config.guess)</parameter> option to
-    will be <command>$LFS_TGT-gcc</command> and the <command>readelf</command>
+    enable <quote>the cross-compilation mode</quote> as we've discussed.
-    tool will be <command>$LFS_TGT-readelf</command>. The kernel headers can
+    The <envar>DESTDIR</envar> variable is used to force installation into
-    be a bit more complicated. Therefore, we take no risks and use
+    the LFS file system.</para>
-    the available configure switch to enforce the correct selection. After
-    the run of <command>configure</command>, check the contents of the
-    <filename>config.make</filename> file in the <filename
-    class="directory">build</filename> directory for all important details.
-    These items highlight an important aspect of the glibc
-    package&mdash;it is very self-sufficient in terms of its build machinery,
-    and generally does not rely on toolchain defaults.</para>
 
     <para>As mentioned above, the standard C++ library is compiled next, followed in
     <xref linkend="chapter-temporary-tools"/> by other programs that must
     be cross-compiled to break circular dependencies at build time.
-    The install step of all those packages uses the
+    The steps for those packages are similar to the steps for glibc.</para>
-    <envar>DESTDIR</envar> variable to force installation
-    in the LFS filesystem.</para>
 
     <para>At the end of <xref linkend="chapter-temporary-tools"/> the native
     LFS compiler is installed. First binutils-pass2 is built,
     in the same <envar>DESTDIR</envar> directory as the other programs,
     then the second pass of gcc is constructed, omitting some
-    non-critical libraries.  Due to some weird logic in gcc's
+    non-critical libraries.</para>
-    configure script, <envar>CC_FOR_TARGET</envar> ends up as
-    <command>cc</command> when the host is the same as the target, but
-    different from the build system. This is why
-    <parameter>CC_FOR_TARGET=$LFS_TGT-gcc</parameter> is declared explicitly
-    as one of the configuration options.</para>
 
     <para>Upon entering the chroot environment in <xref
     linkend="chapter-chroot-temporary-tools"/>,