[virt-tools-list] [virt-viewer PATCH 04/14] po: minimize & canonicalize translations stored in git
Daniel P. Berrangé
berrange at redhat.com
Wed Feb 20 17:50:55 UTC 2019
Similar to the virt-viewer.pot, .po files contain line numbers and file
names identifying where in the source a translatable string comes from.
The source locations in the .po files are thrown away and replaced with
content from the virt-viewer.pot whenever msgmerge is run, so this is not
precious information that needs to be stored in git.
When msgmerge processes a .po file, it will add in any msgids from the
virt-viewer.pot that were not already present. Thus, if a particular msgid
currently has no translation, it can be considered redundant and again
does not need storing in git.
When msgmerge processes a .po file and can't find an exact existing
translation match, it will try todo fuzzy matching instead, marking such
entries with a "# fuzzy" comment to alert the translator to take a
look and either discard, edit or accept the match. Looking at the
existing fuzzy matches in .po files shows that the quality is awful,
with many having a completely different set of printf format specifiers
between the msgid and fuzzy msgstr entry. Fortunately when msgfmt
generates the .gmo, the fuzzy entries are all ignored anyway. The fuzzy
entries could be useful to translators if they were working on the .po
files directly from git, but Virt-Viewer outsourced translation to the
Fedora Zanata system, so keeping fuzzy matches in git is not much help.
Finally, by default msgids are sorted based on source location. Thus, if
a bit of code with translatable text is moved from one file to another,
it may shift around in the .po file, despite the msgid not itself changing.
If the msgids were sorted alphabetically, the .po files would have
stable ordering when code is refactored.
This patch takes advantage of the above observations to canonicalize
and minimize the content stored for .po files in git. Instead of storing
the real .po files, we now store .mini.po files.
The .mini.po files are the same file format as .po files, but have no
source location comments, are sorted alphabetically, and all fuzzy
msgstrs and msgids with no translation are discarded. This cuts the size
of content in the po directory.
Users working from a virt-viewer git checkout who need the full .po files
can run "make update-po", which merges the virt-viewer.pot and .mini.po
file to create a .po file containing all the content previously stored
in git.
Conversely if a full .po file has been modified, for example, by
downloading new content from Zanata, the .mini.po files can be updated
by running "make update-mini-po". The resulting diffs of the .mini.po
file will clearly show the changed translations without any of the noise
that previously obscured content. Being able to see content changes
clearly actually identified a bug in the zanata python client where it
was adding bogus "fuzzy" annotations to many messages:
https://bugzilla.redhat.com/show_bug.cgi?id=1564497
Users working from virt-viewer releases should not see any difference in
behaviour, since the tarballs only contain the full .po files, not the
.mini.po files.
As an added benefit, generating tarballs with "make dist", will no
longer cause creation of dirty files in git, since it won't touch the
.mini.po files, only the .po files which are no longer kept in git.
The languages are minimized in the following commit since it is a
large mechanical process.
Signed-off-by: Daniel P. Berrangé <berrange at redhat.com>
---
build-aux/minimize-po.pl | 37 +++++++++++++++++++++++++
po/Makefile.am | 30 +++++++++++----------
po/README.md | 58 +++++++++++++++++++++++++++++++++-------
3 files changed, 101 insertions(+), 24 deletions(-)
create mode 100755 build-aux/minimize-po.pl
diff --git a/build-aux/minimize-po.pl b/build-aux/minimize-po.pl
new file mode 100755
index 0000000..497533a
--- /dev/null
+++ b/build-aux/minimize-po.pl
@@ -0,0 +1,37 @@
+#!/usr/bin/perl
+
+my @block;
+my $msgstr = 0;
+my $empty = 0;
+my $unused = 0;
+my $fuzzy = 0;
+while (<>) {
+ if (/^$/) {
+ if (!$empty && !$unused && !$fuzzy) {
+ print @block;
+ }
+ @block = ();
+ $msgstr = 0;
+ $fuzzy = 0;
+ push @block, $_;
+ } else {
+ if (/^msgstr/) {
+ $msgstr = 1;
+ $empty = 1;
+ }
+ if (/^#.*fuzzy/) {
+ $fuzzy = 1;
+ }
+ if (/^#~ msgstr/) {
+ $unused = 1;
+ }
+ if ($msgstr && /".+"/) {
+ $empty = 0;
+ }
+ push @block, $_;
+ }
+}
+
+if (@block && !$empty && !$unused) {
+ print @block;
+}
diff --git a/po/Makefile.am b/po/Makefile.am
index f32537d..4950cbe 100644
--- a/po/Makefile.am
+++ b/po/Makefile.am
@@ -2,7 +2,7 @@ DOMAIN = $(PACKAGE_NAME)
COPYRIGHT_HOLDER = The Libvirt authors
MSGID_BUGS_ADDRESS = https://libvirt.org/bugs.html
-MAINTAINERCLEANFILES = $(GMOFILES) $(POTFILE)
+MAINTAINERCLEANFILES = $(GMOFILES) $(POTFILE) $(POFILES)
LANGS := \
af am anp ar as ast bal be bg \
@@ -48,27 +48,28 @@ SED_PO_FIXUP_ARGS = \
-e "s|Copyright (C) YEAR|Copyright (C) $$(date +'%Y')|" \
$(NULL)
-
-# Although they're in EXTRA_DIST, we still need to
-# copy these again, because update-gmo will change
-
-# their content, and dist-hook runs after the
-# things in EXTRA_DIST are copied.
-dist-hook: $(GMOFILES)
- cp -f $(POTFILE) $(distdir)/
- cp -f $(POFILES) $(distdir)/
- cp -f $(GMOFILES) $(distdir)/
+all: update-po
update-po: $(POFILES)
update-gmo: $(GMOFILES)
+update-mini-po: $(POTFILE)
+ for lang in $(LANGS); do \
+ echo "Minimizing $$lang content" && \
+ $(MSGMERGE) --no-location --no-fuzzy-matching --sort-output \
+ $(srcdir)/$$lang.po $(POTFILE) | \
+ $(SED) $(SED_PO_FIXUP_ARGS) | \
+ $(PERL) $(top_srcdir)/build-aux/minimize-po.pl > \
+ $(srcdir)/$$lang.mini.po ; \
+ done
+
push-pot: $(POTFILE)
zanata push --push-type=source
pull-po: $(POTFILE)
zanata pull --create-skeletons
- $(MAKE) update-po
+ $(MAKE) update-mini-po
$(MAKE) update-gmo
$(POTFILE): POTFILES $(POTFILE_DEPS)
@@ -77,8 +78,9 @@ $(POTFILE): POTFILES $(POTFILE_DEPS)
$(SED) $(SED_PO_FIXUP_ARGS) < $@-t > $@
rm -f $@-t
-$(srcdir)/%.po: $(POTFILE)
- $(MSGMERGE) --backup=off --no-fuzzy-matching --update $@ $(POTFILE)
+$(srcdir)/%.po: $(srcdir)/%.mini.po $(POTFILE)
+ $(MSGMERGE) --no-fuzzy-matching $< $(POTFILE) | \
+ $(SED) $(SED_PO_FIXUP_ARGS) > $@
$(srcdir)/%.gmo: $(srcdir)/%.po
rm -f $@ $@-t
diff --git a/po/README.md b/po/README.md
index 20ebf83..a9c8e94 100644
--- a/po/README.md
+++ b/po/README.md
@@ -7,18 +7,39 @@ and file formats, in combination with the Zanata web service.
Source repository
=================
-The virt-viewer GIT repository stores the master "virt-viewer.pot" file and
-full "po" files for translations. The master "virt-viewer.pot" file can be
-re-generated using
+The virt-viewer GIT repository does NOT store the master "virt-viewer.pot"
+file, nor does it store full "po" files for translations. The master
+"virt-viewer.pot" file can be generated at any time using
make virt-viewer.pot
-The full po files can have their source locations and msgids updated using
+The translations are kept in minimized files that are the same file format
+as normal po files but with all redundant information stripped and messages
+re-ordered. The key differences between the ".mini.po" files in GIT and the
+full ".po" files are
+
+ - msgids with no current translation are omitted
+ - msgids are sorted in alphabetical order not source file order
+ - msgids with a msgstr marked "fuzzy" are discarded
+ - source file locations are omitted
+
+The full po files can be created at any time using
make update-po
-Normally these updates are only done when either refreshing translations from
-Zanata, or when creating a new release.
+This merges the "virt-viewer.pot" with the "$LANG.mini.po" for each language,
+to create the "$LANG.po" files. These are included in the release archives
+created by "make dist".
+
+When a full po file is updated, changes can be propagated back into the
+minimized po files using
+
+ make update-mini-po
+
+Note, however, that this is generally not something that should be run by
+developers normally, as it is triggered by 'make pull-po' when refreshing
+content from Zanata.
+
Zanata web service
==================
@@ -30,9 +51,26 @@ translation team using the Zanata web service:
As such, changes to translations will generally NOT be accepted as patches
directly to virt-viewer GIT. Any changes made to "$LANG.mini.po" files in
-virt-viewer GIT will be overwritten and lost the next time content is imported
-from Zanata.
+virt-viewer GIT will be overwritten and lost the next time content is
+imported from Zanata.
The master "virt-viewer.pot" file is periodically pushed to Zanata to provide
-the translation team with content changes. New translated text is then
-periodically pulled down from Zanata to update the po files.
+the translation team with content changes, using
+
+ make push-pot
+
+New translated text is then periodically pulled down from Zanata to update the
+minimized po files, using
+
+ make pull-po
+
+Sometimes the translators make mistakes, most commonly with handling printf
+format specifiers. The "pull-po" command re-generates the .gmo files to try to
+identify such mistakes. If a mistake is made, the broken msgstr should be
+deleted in the local "$LANG.mini.po" file, and the Zanata web interface used
+to reject the translation so that the broken msgstr isn't pulled down next time.
+
+After pulling down new content the diff should be examined to look for any
+obvious mistakes that are not caught automatically. There have been bugs in
+Zanata tools which caused messges to go missing, so pay particular attention to
+diffs showing deletions where the msgid still exists in virt-viewer.pot
--
2.20.1
More information about the virt-tools-list
mailing list