RFR(S): 8213698: Improve devkit creation and add support for linux/ppc64/ppc64le/s390x

Magnus Ihse Bursie magnus.ihse.bursie at oracle.com
Thu Nov 22 16:09:55 UTC 2018



On 2018-11-22 11:27, Volker Simonis wrote:
> On Tue, Nov 20, 2018 at 1:46 PM Magnus Ihse Bursie
> <magnus.ihse.bursie at oracle.com> wrote:
>> Hi Volker,
>>
>> Glad to see you fix and clean up the devkit scripts! And you're more
>> than welcome to add documentation to the building.md.
>>
> OK, I've added a section about devkits to build.{md,html}. Please have
> a look and let me know what you think. Any comments are welcome :)
>
> http://cr.openjdk.java.net/~simonis/webrevs/2018/8213698.v2/
Looks good, all changes below, and the additional readme text!

A hint (you don't need to update your changes): if you have long links 
in markdown, instead of having to line break them
like this (and move a word to the new line)

described in [section 3.4 of the GNU
Autobook](https://sourceware.org/autobook/autobook/autobook_17.html). If


you can put a line break right before the URL, after the opening 
parenthesis, like this:

described in [section 3.4 of the GNU  Autobook]( https://sourceware.org/autobook/autobook/autobook_17.html). If


It usually improves readability of the markdown file.

/Magnus
>
> Please find my comments to the other changes inline
>
>> I think all your changes are good, and can go in as-is, but I see some
>> further potential for cleanup, so if you feel like some more fixes while
>> you're at it, please go ahead. Otherwise you can ignore the rest of what
>> I'm writing. :)
>>
>>> make cross_compile_target="ppc64-linux-gnu" BASE_OS=Fedora BASE_OS_VERSION=17
>> For consistency, maybe we should rename cross_compile_target to TARGET?
>> At least as written on the make command line; you can keep the
>> cross_compile_target name in the Makefile itself, and only assign to it
>> from TARGET.
>>
> I've renamed it to TARGETS because it can actually take a whole list
> of targets (and also to distinguish it from the TARGET parameter used
> in Tools.gmk which only accepts a single TARGET).
>
>>> OEL_URL :=
>>> https://archives.fedoraproject.org/pub/archive/$(FEDORA_TYPE)/releases/$(BASE_OS_VERSION)/Everything/$(ARCH)/os/Packages/
>> Maybe it's time to rename this to something more distro-agnostic than
>> OEL_URL...
>>
> Renamed it to BASE_URL
>
>>> PREFIX=$(RESULT)/$@-to-$$p
>> I really like this! Devkits are much more better described as X-to-Y
>> (even if that was not originally the case, as Erik said).
>>
> Thanks :)
>
>>> Notice that we have to build "ccache" for each target because ccache
>>> will be installed into the directory specified by "--prefix" at
>>> configure time and this is now different for every target. However
>>> that's not a big problem, because the time for compiling ccache is
>>> negligible compared to the download time of the RPMs and the build
>>> time of GCC.
>> Is it still worth the effort of compiling ccache? At least in Oracle,
>> we're not using it anymore. OTOH, if it works, we can just keep it
>> there. It's no big cost.
>>
> I'm not using it either, but I leave it in for now.
>
>>> - Cleanup arch selection:
>>>
>>>    ifeq ($(ARCH),x86_64)
>>> -  RPM_ARCHS := x86_64 noarch
>>> +  RPM_ARCHS := $(ARCH) noarch
>>>      ifeq ($(BUILD),$(HOST))
>>>        ifeq ($(TARGET),$(HOST))
>>>          # When building the native compiler for x86_64, enable mixed mode.
>>> @@ -199,7 +206,7 @@
>>>        endif
>>>      endif
>>>    else ifeq ($(ARCH),i686)
>>> -  RPM_ARCHS := i386 i686 noarch
>>> +  RPM_ARCHS := $(ARCH) i386 noarch
>>>    else ifeq ($(ARCH), armhfp)
>> Looks like you could always to RPM_ARCHS := $(ARCH) noarch, and then use
>> += for the few platforms that need additionals RPM archs.
>>
> Done.
>
>> There was something more I thought of when I reviewed your changes, but
>> it slipped my mind now.
>>
> There's still room for improvements :) For example the GCC version
> could be made configurable, as well as the OEL version. But I leave
> that to the person who updates the compiler the next time :)
>
>> /Magnus
>>
>>
>> On 2018-11-12 12:19, Volker Simonis wrote:
>>> Hi,
>>>
>>> can I please have a review for the following change which ads support
>>> for linux/ppc64/ppc64le/s390x devkits and hopefully improves the
>>> creation of devkits a little bit :)
>>>
>>> http://cr.openjdk.java.net/~simonis/webrevs/2018/8213698/
>>> https://bugs.openjdk.java.net/browse/JDK-8213698
>>>
>>> With these changes it becomes possible to say any of the following:
>>>
>>> make cross_compile_target="ppc64le-linux-gnu s390x-linux-gnu" BASE_OS=Fedora
>>> make cross_compile_target="ppc64-linux-gnu" BASE_OS=Fedora BASE_OS_VERSION=17
>>> make onlytars cross_compile_target="ppc64-linux-gnu ppc64le-linux-gnu
>>> s390x-linux-gnu"
>>>
>>> and get the following devkits under "build/devkit/result":
>>>
>>> sdk-x86_64-unknown-linux-gnu-to-ppc64le-linux-gnu-20181112.tar.gz
>>> sdk-x86_64-unknown-linux-gnu-to-ppc64-linux-gnu-20181112.tar.gz
>>> sdk-x86_64-unknown-linux-gnu-to-s390x-linux-gnu-20181112.tar.gz
>>> x86_64-unknown-linux-gnu-to-ppc64le-linux-gnu/
>>> x86_64-unknown-linux-gnu-to-ppc64-linux-gnu/
>>> x86_64-unknown-linux-gnu-to-s390x-linux-gnu/
>>>
>>> Below you can find a more detailed description of the various changes.
>>> Once we've discussed and agreed on the changes I'd like to add a small
>>> documentation about how to build and use devkits to
>>> "doc/building.{md,html}" which describes the creation and usage of
>>> devkits. This documentation should be right at the top of the
>>> "Cross-compiling" section which is quite complex now. It was not clear
>>> to me until yet how trivial the creation and usage of a devkit can be
>>> :)
>>>
>>>
>>> - The changes required for supporting linux/ppc64/ppc64le/s390x are trivial:
>>>
>>> make/devkit/Tools.gmk
>>>
>>> +ifneq ($(filter ppc64 ppc64le s390x, $(ARCH)), )
>>> +  # We only support 64-bit on these platforms anyway
>>> +  CONFIG += --disable-multilib
>>> +endif
>>>
>>> This is required to prevent building of multilib toolchains which
>>> arent't needed anyway. The problem with the multilib toolchain build
>>> is that it requires some special 32-bit headers which arn't installed
>>> by default from the current RPM list.
>>>
>>> - The following change allows users to choose the version of Fedora
>>> which is used to create the sysroot environment by setting
>>> "BASE_OS_VERSION" (with "27" being the default). This works "BASE_OS"
>>> will be set to "Fedora" (as opposed to "Fedora27" before). Notice that
>>> older Fedora versions have a sligthly different download URL:
>>>
>>> make/devkit/Tools.gmk
>>>
>>>    ifeq ($(BASE_OS), OEL6)
>>>      OEL_URL := http://yum.oracle.com/repo/OracleLinux/OL6/4/base/$(ARCH)/
>>>      LINUX_VERSION := OEL6.4
>>> -else ifeq ($(BASE_OS), Fedora27)
>>> -  ifeq ($(ARCH), aarch64)
>>> +else ifeq ($(BASE_OS), Fedora)
>>> +  DEFAULT_OS_VERSION := 27
>>> +  ifeq ($(BASE_OS_VERSION), )
>>> +    BASE_OS_VERSION := $(DEFAULT_OS_VERSION)
>>> +  endif
>>> +  ifeq ($(filter x86_64 armhfp, $(ARCH)), )
>>>        FEDORA_TYPE=fedora-secondary
>>>      else
>>>        FEDORA_TYPE=fedora/linux
>>>      endif
>>> -  OEL_URL := https://dl.fedoraproject.org/pub/$(FEDORA_TYPE)/releases/27/Everything/$(ARCH)/os/Packages/
>>> -  LINUX_VERSION := Fedora 27
>>> +  ARCHIVED := $(shell [ $(BASE_OS_VERSION) -lt $(DEFAULT_OS_VERSION)
>>> ] && echo true)
>>> +  ifeq ($(ARCHIVED),true)
>>> +    OEL_URL :=
>>> https://archives.fedoraproject.org/pub/archive/$(FEDORA_TYPE)/releases/$(BASE_OS_VERSION)/Everything/$(ARCH)/os/Packages/
>>> +  else
>>> +    OEL_URL :=
>>> https://dl.fedoraproject.org/pub/$(FEDORA_TYPE)/releases/$(BASE_OS_VERSION)/Everything/$(ARCH)/os/Packages/
>>> +  endif
>>> +  LINUX_VERSION := Fedora $(BASE_OS_VERSION)
>>>    else
>>>      $(error Unknown base OS $(BASE_OS))
>>>    endif
>>>
>>> - Enable the creation of several different devkits at once (e.g. 'make
>>>    cross_compile_target="ppc64-linux-gnu ppc64le-linux-gnu
>>> s390x-linux-gnu"') or one after another but all into the same
>>> 'build/devkit/result' directory. The result directory will contain
>>> $HOST-to-$TARGET sub-directories with the corresponding devkits:
>>>
>>> make/devkit/Makefile
>>>
>>> -submakevars = HOST=$@ BUILD=$(me) \
>>> -    RESULT=$(RESULT) PREFIX=$(RESULT)/$@ \
>>> -    OUTPUT_ROOT=$(OUTPUT_ROOT)
>>> +submakevars = HOST=$@ BUILD=$(me) RESULT=$(RESULT) OUTPUT_ROOT=$(OUTPUT_ROOT)
>>> +
>>>    $(host_platforms) :
>>>           @echo 'Building compilers for $@'
>>>           @echo 'Targets: $(target_platforms)'
>>>           for p in $(filter $@, $(target_platforms)) $(filter-out $@,
>>> $(target_platforms)); do \
>>> -         $(MAKE) -f Tools.gmk download-rpms $(submakevars) TARGET=$$p && \
>>> +         $(MAKE) -f Tools.gmk download-rpms $(submakevars) \
>>> +              TARGET=$$p PREFIX=$(RESULT)/$@-to-$$p && \
>>>             $(MAKE) -f Tools.gmk all $(submakevars) \
>>> -             TARGET=$$p || exit 1 ; \
>>> +              TARGET=$$p PREFIX=$(RESULT)/$@-to-$$p && \
>>> +         $(MAKE) -f Tools.gmk ccache $(submakevars) \
>>> +              TARGET=$@ PREFIX=$(RESULT)/$@-to-$$p
>>> BUILDDIR=$(OUTPUT_ROOT)/$@/$$p || exit 1 ; \
>>>           done
>>> -       @echo 'Building ccache program for $@'
>>> -       $(MAKE) -f Tools.gmk ccache $(submakevars) TARGET=$@
>>>           @echo 'All done"'
>>>
>>> Notice that we have to build "ccache" for each target because ccache
>>> will be installed into the directory specified by "--prefix" at
>>> configure time and this is now different for every target. However
>>> that's not a big problem, because the time for compiling ccache is
>>> negligible compared to the download time of the RPMs and the build
>>> time of GCC.
>>>
>>>    define Mktar
>>> -  $(1)_tar = $$(RESULT)/sdk-$(1)-$$(today).tar.gz
>>> -  $$($(1)_tar) : PLATFORM = $(1)
>>> -  TARFILES += $$($(1)_tar)
>>> -  $$($(1)_tar) : $(1) $$(shell find $$(RESULT)/$(1))
>>> +  $(1)-to-$(2)_tar = $$(RESULT)/sdk-$(1)-to-$(2)-$$(today).tar.gz
>>> +  $$($(1)-to-$(2)_tar) : PLATFORM = $(1)-to-$(2)
>>> +  TARFILES += $$($(1)-to-$(2)_tar)
>>> +  $$($(1)-to-$(2)_tar) : $$(shell find $$(RESULT)/$(1)-to-$(2) -type f)
>>>    endef
>>>
>>> -$(foreach p,$(host_platforms),$(eval $(call Mktar,$(p))))
>>> +$(foreach p,$(host_platforms),$(foreach t,$(target_platforms),$(eval
>>> $(call Mktar,$(p),$(t)))))
>>>
>>> make/devkit/Tools.gmk
>>>
>>> -PATHEXT = $(RESULT)/$(BUILD)/bin:
>>> +PATHEXT = $(PREFIX)/bin:
>>>
>>>
>>> - Various small cleanups
>>>
>>> make/devkit/Tools.gmk
>>>
>>> - Don't set  "RESULT" and "PREFIX" in Tools.gmk because the values are
>>> overridden by the settings in the calling Makefile anyway:
>>>
>>>    # Define directories
>>> -RESULT := $(OUTPUT_ROOT)/result
>>>    BUILDDIR := $(OUTPUT_ROOT)/$(HOST)/$(TARGET)
>>> -PREFIX := $(RESULT)/$(HOST)
>>>    TARGETDIR := $(PREFIX)/$(TARGET)
>>>
>>> - Cleanup arch selection:
>>>
>>>    ifeq ($(ARCH),x86_64)
>>> -  RPM_ARCHS := x86_64 noarch
>>> +  RPM_ARCHS := $(ARCH) noarch
>>>      ifeq ($(BUILD),$(HOST))
>>>        ifeq ($(TARGET),$(HOST))
>>>          # When building the native compiler for x86_64, enable mixed mode.
>>> @@ -199,7 +206,7 @@
>>>        endif
>>>      endif
>>>    else ifeq ($(ARCH),i686)
>>> -  RPM_ARCHS := i386 i686 noarch
>>> +  RPM_ARCHS := $(ARCH) i386 noarch
>>>    else ifeq ($(ARCH), armhfp)
>>>
>>> - Don't create 'devkit.info' unconditinally. Only build it as part of
>>> the "all" target invocation. This prevents the creation of a
>>> 'devkit.info' with the wrong "BASE_OS_VERSION" if the Makefile is
>>> invoked several times with different "BASE_OS_VERSION" values:
>>>
>>> -$(PREFIX)/devkit.info: FRC
>>> +$(PREFIX)/devkit.info:
>>>           @echo 'Creating devkit.info in the root of the kit'
>>>           rm -f $@
>>>           touch $@
>>> @@ -611,7 +623,4 @@
>>>    # this is only built for host. so separate.
>>>    ccache : $(ccache)
>>>
>>> -# Force target
>>> -FRC:
>>>
>>> - put the base directory of the devkits into the tar archives. I don't
>>> like tar archives which don't have a single top-level directory and
>>> expand right into the current working directory :)
>>>
>>> make/devkit/Makefile
>>>
>>>    %.tar.gz :
>>>           @echo 'Creating compiler package $@'
>>> -       cd $(RESULT)/$(PLATFORM) && tar -czf $@ *
>>> +       cd $(RESULT) && tar -czf $@ $(PLATFORM)/*
>>>           touch $@



More information about the build-dev mailing list