Porting Mozilla to *BSD
How to Keep Up with Fast Moving Projects and Upstream Patches
Landry Breuil <landry@openbsd.org>
EuroBSDCon 2013 / Sep,28 / Malta
OpenBSD
As a project
- CVS (every time you use it, a kitten dies)
- 6-month release schedule, 1-year support
- Evolution, not revolution
- Hostility towards gplv3 (ie old toolchain in base)
- "Shut up and hack" & "eat your own dogfood"
- ~120 active developers, ~30/40 'porters'
- ~~50 commits/day
- Regular hackathons all around the world
As a desktop/laptop OS
- use -CURRENT
- binary packages / pkg_add -u since 2005
- Xfce 4.10, GNOME 3.8, KDE 4.10 (wip)
- Chromium, LibreOffice, Mplayer, games...
- suspend/resume via our own ACPI implem
- DRM/KMS on intel/ati, recent MESA, gallium/llvmpipe (wip)
- rthreads!
- no udev, maybe someday?
Mozilla, the project
Not only firefox
- Gecko, the engine (xulrunner)
- Thunderbird, still rocking w/ lightning + sogo extension
- Seamonkey, the venerable Mozilla suite
- Instantbird, the IM client
- Firefox Sync / Mozilla Persona
- Addons
- FirefoxOS
Infrastructure/Community
- ~thousand contributors, ~400 active monthly
- Thousands of translators/reps/support/QA..
- ~400 checkins/day
- 6000 builds/day, 50k unittests/day
- 5 DCs all around the world
- Thousands of servers/EC2 hosts
- Went from hundred of mirrors in the world to a CDN
- http://blog.mozilla.org/it/
Fast release schedule
In use since Firefox 4.0
Platforms
- Tier1
- Android/arm (gcc)
- Linux/x86(-64) (gcc) (in // with distrib maintainers)
- OSX/x86(-64) (clang)
- Windows/x86 (msvc)
- Tier2/Tier3.. the others (os/2 or aix, anyone ?)
- Crazy people like TenFourFox, Classilla or Bezilla
- And us! (even on sparc64, thx to martin@NetBSD!)
Mozilla, the tools
Mercurial
Queues
hg qpop, hg qseries, hg qnew, hg qpush, hg qfinish...
Branches / inbound
Sheriffs watch the tree, star failures, and handle merges
Project repositories (fx-team, build-system, b2g-inbound, birch, alder..)
Bugzilla a.k.a. bmo
- Everything is handled there (code, hw, netops, accounts, releng..)
- No commit without a bug #
- Splinter for code review
- User / Product dashboards
- Integration with MQ: bzexport & qimportbz
- Integration with Mercurial: Close bugs from commit msgs
- Integration with MUA via email_in.pl / BzAPI via REST
Workflow
- Find breakage, cry
- File a bug
- Patch against m-c
- r?
- repeat previous until r+
- checkin-needed / hg push
- (watch the tree)
- approval-{aurora,beta,release}? (optional)
- profit!
Tbpl (a.k.a. TinderboxPushLog)
http://tbpl.mozilla.org
Try
dawn:~/src/mozilla-central/ $hg qseries
bug-844818-separate-alsa-pulse-webrtc
bug-844430-libsctp-openbsd
bug-807492-webrtc-bsd
bug-807492-webrtc-openbsd-fixes
try
dawn:~/src/mozilla-central/ $hg log -l1 | grep summary
summary: try: -b do -p all -u all -t all
dawn:~/src/mozilla-central/ $hg push -f ssh://hg.mozilla.org/try
Runs all tests on all platforms with mozilla-central tip + my patches
Commit bit
- File a bug!
- Level 1: access to try, need 1 voucher
- SSH key / ldap account / committer's agreement
- Keep nagging ppl with commit bit to push your checkin-needed patches
- Level 3: full access, need 2 vouchers from module owners
- First bug filed feb-2010, Got L1 apr-2011, L3 mid-2012
Not only firefox
devel/xulrunnerdevel/spidermonkeylang/spidermonkey (don't ask..)mail/mozilla-thunderbird{,-i18n}productivity/sunbirdwww/mozilla-firefox{,-i18n}www/seamonkey{,-i18n}www/firefox36www/fennec
History
- mozilla-19980728, not working
- 2002/2003: initial Mozilla 1.1 work, a.out only (jolan@)
- May 2003: Mozilla 1.4a/Firebird 0.6 runs (i386/sparc64/alpha) ! (wilfried@)
- Sept 2003: Flamewar on ports@ about which mozilla should be default
- January 2004: Thunderbird 0.4
- February 2004: Firefox 0.8
- End of 2005: Firefox 1.5 (bernd@)
History (cont'd)
- End of 2006: Seamonkey 1.0.7 (kurt@)
- March 2007: Firefox 2.0 (martynas@)
- May 2007: Mozilla 1.7 to the Attic/
- Mid-2008: Firefox 3.0
- June 2008: Firefox 3.5
- March 2010: Firefox 3.6 (landry@)
- March 2011: Firefox 4.0!
State in 2010
- TONS of local patches
- No comments in patches
- Maintainer MIA
- Duplicated stuff everywhere
- Diverging stuff everywhere
- I needed lightning in TB so...
- Scratch your own itch!!
Patchy patchy patch
Local customization vs upstreamable patch ?
mozilla.port.mk
- MODULES framework
- Abstracts common things between all 6 users
- ports Makefile greatly shrinked (- 90/100 lines)
- Sets common depends/configure options
- MOZILLA_PROJECT/MOZILLA_BRANCH
- Old dead stuff vs maintained & released stuff
branches workflow in git
- Would have been better as a graph, but i suck at dia(1)
- Fx X.0bY released - work in beta
- merge beta to release when a final/candidate version is released
- merge release to master when commiting to cvs
- merge master to beta to stay in sync wrt cvs
- Fx X.0.Z chemspill - work in release, merge to master & to beta
updating a port
# Change MOZILLA_VERSION, adjust deps
make makesum
make patch
# fix/rm/add patches if needed
make update-patches
make update-plist
make port-lib-depends-check
TEST. FEEDBACK. TEST.
State in 2011
releases/betas are tracked...
patches are being pushed upstream...
let's be proactive and tackle aurora/central!
Buildbot
- http://buildbot.rhaalovely.net
- Dashboard keeping track of all builds w/ logs
- Pull hg tip, apply patches, configure, build, package, upload
- Nightly build of mozilla-aurora, mozilla-central & comm-central
- Produce a mozilla package directly usable for testing
- As close as possible from upstream builds
- Allows to find regression windows
Hardware
- amd64: sun v20z, opteron 252 2.6ghz / 4g (2h30)
- i386: sun x2100, opteron 175 2.2ghz / 2g (1h30)
- sparc64: sun v240, us-IIIi 1Ghz / 5g (9h30)
- macppc: mac mini 1.5ghz / 1g (10 hours!)
- FreeBSD/amd64: bhyve guest @ xeon E3 3.5ghz / 8g (20mn!)
- Xserve ppc g5 on the way (someday)
- + desktops/laptops...
- Needed: beefier HW, moar RAM for objdirs in mfs!
Relationship with upstream
- They start knowing we're here
- I know who to ask when a problem is in a specific area (IRC helps!)
- Not priority platform, but try to not break it (sortof)
- feedback? on important changes
- We need to do the work
- Upstream welcoming++ clean portability patches!!!
- m-c builds unpatched (most of the time) since mid-2012
Relationship between *BSDs
- Know your alter ego
- CC people on patches sent upstream
- Cherrypick selected patches in other ports
- Fight for a common cause
- Could be improved... being more helps !
- Any volunteer for a NetBSD/Dragonfly buildslave ?
Main problems
- Use of
posix linux apis
- Bundled/'forked' libs: cairo, pixman, ICU, libffi, harfbuzz, graphite2, libogg, libvpx, jpeg, png, bz2,...
- Dependency on very latest sqlite/nss/nspr, wonky sqlite options (SECURE_DELETE anyone?)
- --with-system-XXX usually not-so-supported
- *HUGE* code drops (IPC, WebRTC, IonMonkey, ICU...)
- Need newer gcc or clang since Fx 17
- Memory hog when linking libxul.so
- More & more problems with OLD toolchain (ld/gdb)
- Constant babysitting of the buildbot needed
State in 2013
- Up to date, closely following upstream
- HTML5 A/V via GStreamer
- WebGL (works on intel & ati!)
- ESR not considered
- Way less patches, much more documented
- Ports Makefiles sanitized
- Lack of feedback from users....
- What to do with 'old' things ?
Future work
- WebRTC (Fx 26 for BSDs, crash @runtime)
- Electrolysis/IPC/sandboxing
devel/xulrunner/24 ?- Telemetry/FHR
- Breakpad/Profiler
- Push back results from buildbot to tbplv2
←
→
/
#