I just had this error show up from running the AutoPkg recipe, when running autopkg while my shell's current directory was the autopkg Git root.

Error in com.github.autopkg.autopkg: Processor: PkgCreator: Error: Can't find scripts directory: Scripts

The new functionality for PkgCreator to support searching for relative paths also seems to return a path if it finds one in the current working directory (like the "Scripts" dir in the autopkg repo).

The doc string says this function is responsible for searching the search order, so I'm not sure if this was here for a specific reason or to account for a certain case that's not obvious to me. Since I'm not really sure, I changed it in an experimental branch to illustrate:

https://github.com/autopkg/autopkg/tree/PkgCreator-relpaths-ignore-cwd

I also removed the conditional startswith("/") because that's already the condition used to call this function from package(). If this all makes sense, I can merge it in.

Hi, I have an autopkg repo that you may want to add/fork. There is only 1 recipe in there at this point, but will be more coming soon.

https://github.com/derak/autopkg-recipes

Derak

My (two) AutoPkg recipes are available for forking at https://github.com/gerardkok/autopkg-recipes. I intend to add more, but I'm not only a beginner at AutoPkg, but also at git and python, so I don't expect to add new recipes quickly.

Please fork my recipe repo if you find it useful. Thanks!

https://github.com/dankeller/autopkg-recipes

As it is highly unlikely that one would ever use case to meaningfully distinguish between two recipe identifiers, I'm proposing a change to https://github.com/autopkg/autopkg/blob/master/Code/autopkg#L150
to instead read if get_identifer_from_recipe_file(match).lower() == identifier.lower():
This would be for usability when creating 'child' recipes, as discussed here:
https://groups.google.com/forum/#!topic/autopkg-discuss/5icJleU9qo4

I would like to see a possibility to define a version_comparison_key for MunkiInstallsItemsCreator processor in recipe. I've mostly run into this when an installs item needs to be defined with CFBundleVersion instead of CFBundleShortVersionString.

I see that MunkiInstallsItemsCreator is calling makepkginfo -f and there's no options for makepkginfo to do this but the processor could of course modify the installs item dictionary before returning.

I could take a closer look and create a pull request when I have way to do this. Any thoughts?

For https://github.com/jessepeterson/autopkg-recipes. Thanks!

Remove old processors from outdated/replaced recipes, move remaining recipe-specific processors into the recipes repo.

Sometimes a Sparkle feed or some other urlopen() call just hangs because of a server problem. Currently this causes the recipe to hang indefinitely as there's no timeout set.

Harder to determine what would be "correct" for a URLDownloader as some downloads are large and could be slow, but I don't think the Sparkle metadata processor should wait longer than 60 seconds before raising a ProcessorError.

In the long run it would be a good idea to make a base class for any processor that uses urllib/urllib2, at least for getting metadata, for default behavior like this, but that can be another issue further down the line.

Originally, autopkgserver supported creating both 'bundle"-syle and "flat" packages. Now that autopkgserver uses pkgbuild (with productbuild a future possibility), only flat packages are supported. Remove all code associated with creating bundle packages and update autopkg/recipes to have only flat pkg resources.

If the directory ~/Library/AutoPkg/RecipeOverrides/ does not exist when using the make-override verb with autopkg, the process fails. Example: https://gist.github.com/hfike/c84e36d8b2f82eb9b0c2

Manually creating the directory fixes the problem.

Reported by mosen1, Jul 22, 2013 via code.google.com/p/macautopkg

What steps will reproduce the problem?

1. Run PkgUnpacker on flat package containing a nested (m)pkg.
2. The nested package isn't unpacked properly (due to xar not caring abo?)

The example package is the Centrify DirectControl 5.1.1 agent. This package is not available for public download but basically it is a flat package containing another (m?)package. When using xar you get the flat package content but the nested package isn't unpacked.

pkgutil --expand does not exhibit the same issue (the scripts are unpacked inside the nested package).

If it's feasible to use pkgutil in another processor then that might be an option.

Would you please fork my recipes available here:
https://github.com/hjuutilainen/autopkg-recipes

Thanks!

Hannes

If a recipe path is given by a filename and it has a ParentRecipe, this recipe won't be found unless it happens to be on the search path (for instance, if the command is executed within the same directory and therefore searches the default . cwd path first).

We should be adding the dirname of the child recipe to the search path dynamically when searching for ParentRecipes.

If a recipe includes a ParentRecipe that can't be found for any reason, autopkg reports "No valid recipe for ". The recipe shows up in autopkg list-recipes, but trying to do autopkg info reports:

No valid recipe found for
Nothing downloaded, packaged or imported.

This is unhelpful because the recipe is clearly listed in the list of recipes, but autopkg reports that it can't find it, when the actual error is that the ParentRecipe can't be found or resolved.

It may be helpful to provide a more specific error message indicating that the ParentRecipe is the weak point.

My repo is https://github.com/jleggat/autopkg-recipes

Thanks,
j

Hi,

please add a proxy support in the processor URLDownloader:

The following recipes failed:
GoogleChrome.pkg
Error in com.github.autopkg.pkg.googlechrome: Processor: URLDownloader: Error: Couldn't download https://dl.google.com/chrome/mac/stable/GGRO/googlechrome.dmg: <urlopen error [Errno 65] No route to host>

Nothing downloaded, packaged or imported.

Thanks Alex

Would be nice to be able to package applications using the equivalent of:

pkgbuild --component /path/to/some.app --install-location /Applications

Even, better, to be able to point to the application on a mounted disk image to eliminate the need to copy from the disk image.

Currently autopkgserver/packager.py disallows this because the files on a mounted disk image are not (usually) owned by the user calling AutoPkg.

automunkiimport was designed to be an experiment, where I could push the functionality of autopkg without worrying about breaking the core autopkg tool.

I think autopkg and automunkiimport should be merged.

The Munki-specifc functionality aside, automunkiimport has several features that would be generally useful for autopkg:

1. Recipe search directories
2. Support for recipe lists
3. Support for the "unified" recipe format
4. Support for recipe input overrides
5. More reporting/logging

...and probably some other things I'm forgetting. In the near future we'd like to add support for disacovering and downloading additional recipes.

Here's a thought about one approach:

autopkg processes one or more recipes.

automunkiimport processes one or more Munki-specific recipes, then runs makecatalogs if there have been any imports. Ideally, though, automunkiimport should also mount the Munki repo before it begins.

Merging the functionality ends up having a lot of "if munkimporting: foo; else: bar" crap in various places.

So:

automunkiimport supports recipe lists: these are simply lists of recipes to process in order.
What if there was a "mount the munki repo" recipe and a "run makecatalogs" recipe? Then you could remove a lot of special logic from automunkiimport, and just use a recipe list:

MountMunkiRepo
Firefox
GoogleChrome
MSOffice2011Updates
MakeCatalogs
UnmountRepo

It might be difficult to factor out everything automunkiimport knows about Munki into recipes, but I have some ideas there.

If we can move this direction (merging the non-Munki functionality of automunkiimport into autopkg -- or more practically, factoring out the munki-specific stuff from automunkiimport and calling it the new autopkg tool), it makes it easier to add support for Absolute Manage, Casper, FielWave, etc if anyone wants to.

While FlatPkgUnpackager works, it uses xar. Unfortunately this doesn't extract the Scripts tarball in a package that has one. Plus it's useful to pkg --flatten a package that's been package --expanded

Any interesting in having these in the main tree?

FlatPkgExpander.py
FlatPkgFlattener.py

If attempting to use git repos, or running recipes requiring munki, the corresponding tools need to be installed. Instead of the current errors, autopkg should check whether or not the tools are installed, and give the user clear warnings to install git or the munkitools.

Error when running a recipe requiring munki:
Processor: MunkiImporter: Error: makepkginfo execution failed with error code 2: No such file or directory

It would be nice if we could specify a default for a Processor's input variable:

input_variables = {
    "branch": {
        "required": False,
        "description": "Descriptive text",
        "default": "release"
    }

So that we don't need to implement our own "defaults" logic in every processor.

We should support being able to just add something like a 'scripts' array to a package request dictionary, and they get added via the appropriate mechanism for either bundle or flat packages.

Bundle just requires them being in the right place and executable, for flat the simplest way is via staging them in a 'scripts' directory, and passing --scripts <dir> to pkgbuild. I've tested that these get inserted properly even if a PackageInfo template is used. I don't think flat packages even require the scripts be executable within the archive, but they might as well be.

This would be useful for the AutoPkg recipe: autopkg/recipes#2

Normally running a recipe, we:

• process cli overrides
• verify
• process

RECIPE_CACHE_DIR is only set in autopackager.process(), so if the previous two steps fail, the tool can't use this env to know where to write a receipt.

We could have an initial "prepare_recipe()"-like step where additional env variables like this can be set, and possibly roll the verify() step into this..

autopkg run -k MUNKI_REPO=/tmp/munki munkitools.munki
Processing munkitools.munki...
Traceback (most recent call last):
  File "/usr/local/bin/autopkg", line 1206, in <module>
    sys.exit(main(sys.argv))
  File "/usr/local/bin/autopkg", line 1200, in main
    exit(subcommands[verb]['function'](argv))
  File "/usr/local/bin/autopkg", line 1026, in run_recipes
    autopackager.process_cli_overrides(recipe, cli_values)
  File "/Library/AutoPkg/autopkglib/__init__.py", line 293, in process_cli_overrides
    update_data(self.env, key, value)
  File "/Library/AutoPkg/autopkglib/__init__.py", line 124, in update_data
    a_dict[key] = do_variable_substitution(value)
  File "/Library/AutoPkg/autopkglib/__init__.py", line 121, in do_variable_substitution
    item[key] = do_variable_substitution(value)
  File "/Library/AutoPkg/autopkglib/__init__.py", line 121, in do_variable_substitution
    item[key] = do_variable_substitution(value)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC/objc/_convenience.py", line 246, in __setitem__setObject_forKey_
    self.setObject_forKey_(container_wrap(value), container_wrap(key))
objc.error: NSInternalInconsistencyException - -[__NSCFDictionary setObject:forKey:]: mutating method sent to immutable object

/cc @arubdesu

In trying out Greg's 'auto' tool, I'm finding that since we moved autopkglib's preferences handling to use CFPreferences, that now the Obj-C types retrieved for any supported preferences - NSCFArray, at least - doesn't make plistlib happy when it tries to write these to a plist, and expectedly it raises a TypeError.

Greg, is there any reason you can think of that we couldn't just swap out plistlib for FoundationPlist? It could sit alongside autopkglib as a separate module.

I've just tested that swapping it in at places like https://github.com/autopkg/autopkg/blob/master/Code/auto#L561 seems to fix it.

(migrated from TODO.md)

If I change installs-type key from 'application' to 'bundle' manually in already imported pkginfo-file then autopkg decides to re-import the same item, resulting with unnecessary duplicate item.
This happens for all items that are updated and autoimported to Munki repo by autopkg.
This happens with version 0.2.3 and 0.2.4.

(migrated from TODO.md)

For those keys that are in an Override, would be nice to have a "merging" option.

E.g. for the "pkginfo" key it would be nice not to have to keep the whole pkginfo key in the Override file, in case the pkginfo key changes in the parent recipe.

(migrated from TODO.md)

Howdy. Any interest in having this in the main tree?

GlobFileFinder.py

Please fork my recipe repo available here:

https://github.com/Jaharmi/autopkg_recipes

To fit in with the current recipe names, I’d like it named:

jaharmi-recipes

Thanks!

Offering my meager repo to the autopkg community:
https://github.com/swy/autopkg-recipes

No need for a different name.

Starting to develop some recipes for my organization. Just have Skitch so far.

It uses the AppInfo.py processor that has a pending pull request though, so it's not ready quite yet.

https://github.com/sheagcraig/sheagcraig-recipes.git

Thanks!

(migrated from TODO.md)

Please fork my recipe repo available here:

https://github.com/hansen-m/autopkg-recipes

Thanks!

Now this might just be some oversight on my part but I can't get the FlashPlayerRepackage recipes to work if my CACHE_DIR points to a folder on an external disk.

I've set the CACHE_DIR in autopkg preferences to "/Volumes/external-disk/temp/autopkg-cache" and if I run the recipe with the defaults, I get:

$ autopkg run -v AdobeFlashPlayerRepackage.pkg.recipe
Processing AdobeFlashPlayerRepackage.pkg.recipe...
AdobeFlashURLProvider
AdobeFlashURLProvider: Found URL http://fpdownload.macromedia.com/get/flashplayer/pdc/11.8.800.168/install_flash_player_osx.dmg
URLDownloader
URLDownloader: Item at URL is unchanged.
URLDownloader: Using existing /Volumes/external-disk/temp/autopkg-cache/com.github.autopkg.pkg.FlashPlayerRepackage/downloads/AdobeFlashPlayer.dmg
EndOfCheckPhase
AdobeFlashDmgUnpacker
AdobeFlashDmgUnpacker: Mounted disk image /Volumes/external-disk/temp/autopkg-cache/com.github.autopkg.pkg.FlashPlayerRepackage/downloads/AdobeFlashPlayer.dmg
PkgCreator
PkgCreator: Connecting
PkgCreator: Sending packaging request
PkgCreator: Disconnecting
/Volumes/external-disk/temp/autopkg-cache/com.github.autopkg.pkg.FlashPlayerRepackage/FlashPlayerPkg isn't owned by <myuid>
Failed.
Receipt written to /Volumes/external-disk/temp/autopkg-cache/com.github.autopkg.pkg.FlashPlayerRepackage/receipts/AdobeFlashPlayerRepackage.pkg-receipt-20130927-092147.plist

The following recipes failed:
    AdobeFlashPlayerRepackage.pkg.recipe
        Error in com.github.autopkg.pkg.FlashPlayerRepackage: Processor: PkgCreator: Error: /Volumes/external-disk/temp/autopkg-cache/com.github.autopkg.pkg.FlashPlayerRepackage/FlashPlayerPkg isn't owned by <myuid>

Nothing downloaded, packaged or imported.

The directory in question is owned by me. However, if I override the CACHE_DIR setting and set it to default, everything works perfectly:

$ autopkg run -v -k CACHE_DIR=~/Library/AutoPkg/Cache AdobeFlashPlayerRepackage.pkg.recipe
Processing AdobeFlashPlayerRepackage.pkg.recipe...
AdobeFlashURLProvider
AdobeFlashURLProvider: Found URL http://fpdownload.macromedia.com/get/flashplayer/pdc/11.8.800.168/install_flash_player_osx.dmg
URLDownloader
URLDownloader: Item at URL is unchanged.
URLDownloader: Using existing /Users/<username>/Library/AutoPkg/Cache/com.github.autopkg.pkg.FlashPlayerRepackage/downloads/AdobeFlashPlayer.dmg
EndOfCheckPhase
AdobeFlashDmgUnpacker
AdobeFlashDmgUnpacker: Mounted disk image /Users/<username>/Library/AutoPkg/Cache/com.github.autopkg.pkg.FlashPlayerRepackage/downloads/AdobeFlashPlayer.dmg
PkgCreator
PkgCreator: Connecting
PkgCreator: Sending packaging request
PkgCreator: Disconnecting
Receipt written to /Users/<username>/Library/AutoPkg/Cache/com.github.autopkg.pkg.FlashPlayerRepackage/receipts/AdobeFlashPlayerRepackage.pkg-receipt-20130927-092552.plist

The following packages were built:
    Identifier               Version          Pkg path                        
    ----------               -------          --------                        
    com.adobe.pkg.FlashPlayer 11.8.800.168     /Users/<username>/Library/AutoPkg/Cache/com.github.autopkg.pkg.FlashPlayerRepackage/AdobeFlashPlayer-11.8.800.168.pkg

Let me know if I'm doing something stupid here... :)

Hannes

MunkiImporter needs to check if a previous application is installed, so it checks the 'all' catalog in Munki against the freshly generated pkginfo (from makepkginfo). This is line 387 in Code/autopkglib/MunkiImporter.py. However, the environment pkginfo isn't merged into that pkginfo until after the check for the app, on line 404.

The effect of this is that, if a pkginfo has a manually added installs key in a Processor before (and thus in the environment pkginfo) and the package has a unique installer hash item (say because it is re-generated every autopkg run, then MunkiImporter will not be catching the alread packaged package in the Munki catalog (because the installs key hasn't yet been added when it checks).

Hope that makes sense. The solution (I think) is to move block at line 404 (if "pkginfo" in self.env:) to before the existing line 387 (matchingitem = self.findMatchingItemInRepo(pkginfo)).

Right now I have an autopkg that continues to add packages (using the __1, __2, etc) naming convention to autopkg, despite it already being packaged.

Thanks, and sorry for the winded explanation.

please fork my repo at http://github.com/keeleysam/recipes

Hi,

i tried the 0.2.5. release unter two 10.6.8 server installations and both failed.

vmserver:RecipeOverrides virt$ autopkg list-recipes
Traceback (most recent call last):
File "/usr/local/bin/autopkg", line 1247, in
sys.exit(main(sys.argv))
File "/usr/local/bin/autopkg", line 1241, in main
exit(subcommands[verb]'function')
File "/usr/local/bin/autopkg", line 848, in list_recipes
if valid_recipe(match):
File "/usr/local/bin/autopkg", line 102, in valid_recipe
return (valid_plist_with_keys(filename, ["Input", "Process"])
File "/usr/local/bin/autopkg", line 88, in valid_plist_with_keys
recipe_plist = FoundationPlist.readPlist(filename)
File "/Library/AutoPkg/FoundationPlist/FoundationPlist.py", line 72, in readPlist
plistData, NSPropertyListMutableContainersAndLeaves, None, None))
ValueError: too many values to unpack

Taking a quick look at the code (def get_processor(processor_name, recipe=None) of autopkg/Code/autopkglib/__init__.py:438) it looks like Processors are only found in the autopkglib directory and recipe directory. I happen to have a number of recipes that share common Processors (see 1, 2, 3, and 4 for an example). Is there any way one can have a directory in a RecipeRepo for shared processors? Something like:

./RecipeDir1/Recipe1.munki.recipe
./RecipeDir2/Recipe2.munki.recipe
./SharedProcessors/myproc.py

Thanks! For everything: autopkg rocks!

To discuss support for versioning AutoPkg and specifying a minimum required version in a recipe.

A couple commits to illustrate very basic support:

• autopkglib support for setting an env, AUTOPKG_VERSION, to the value of the Version key in version.plist: 82c57a7
• AutoPkg recipe update to support it: be94692

This would probably mean that new releases would be frequent, whenever new recipe-related functionality is introduced to support new and updated recipes. A build/push script could be written to automate the process of making a new tag, building a new package, formatting the version-specific release notes from a master CHANGELOG.md file, incrementing to the next version in version.plist, etc.

I imagine this would make the '1.2.3' version numbering much less likely to follow the conventions of SemVer, but with frequent releases, not sure it really matters (?).

It might be nice to have an additional auto-incrementing "build" value appended to this version in the same way that Munki's pkg build script handles this. The AutoPkg building from a source zipfile would mean some additional work would be needed to derive this value, or otherwise optionally instead do a git clone, where it's easier to have direct access to the number of commits.

The package creation workflow should allow for using a cert to create signed packages.

Creating an issue to track the discussion of this.

autopkg run Firefox

autopkg check Firefox

autopkg add-recipes https://github.com/keeleysam/recipes.git

etc.

Homebrew does a neat thing if you try to an install a formula that doesn't yet exist, by searching open pull requests.

Since we can probably usually figure out the upstream remote for a recipe repo, we can use the GitHub API to search issues by a known label like "broken_recipe", so that if there's an error when running a recipe, we can check to see if it's already known (or at least reported by someone) to be broken and report it to the user in the run output.

The GitHub API currently allows for 60 anonymous requests per hour.

Hi, requesting to my Recipe repo to the Autopkg org.
https://github.com/nmcspadden/nmcspadden-recipes

It's only got one item in it so far, but I'll probably be adding more things to it in the future. Thanks!

Apologies but i am following the getting started on the Wiki but not using Munki.

I am getting error. I haven't created any autopkgserver of find documentation in the wiki as to what this might point to.

I would appreciate a hand if possible.

The following recipes failed:
Firefox.pkg
Error in com.github.autopkg.pkg.Firefox_EN: Processor: PkgCreator: Error: Couldn't connect to autopkgserver: No such file or directory

Full verbose of command below

autopkg run -v Firefox.pkg
Processing Firefox.pkg...
MozillaURLProvider
MozillaURLProvider: Found URL http://ftp.mozilla.org/pub/mozilla.org//firefox/releases/latest/mac/en-US/Firefox%2026.0.dmg
URLDownloader
URLDownloader: Storing new Last-Modified header: Thu, 05 Dec 2013 17:54:42 GMT
URLDownloader: Storing new ETag header: "57af9b-2d7ff56-4eccd3961bc80"
URLDownloader: Downloaded /Users/tkimpton/Library/AutoPkg/Cache/com.github.autopkg.pkg.Firefox_EN/downloads/Firefox.dmg
EndOfCheckPhase
AppDmgVersioner
AppDmgVersioner: Mounted disk image /Users/tkimpton/Library/AutoPkg/Cache/com.github.autopkg.pkg.Firefox_EN/downloads/Firefox.dmg
AppDmgVersioner: BundleID: org.mozilla.firefox
AppDmgVersioner: Version: 26.0
PkgRootCreator
PkgRootCreator: Created /Users/tkimpton/Library/AutoPkg/Cache/com.github.autopkg.pkg.Firefox_EN/Firefox
PkgRootCreator: Created /Users/tkimpton/Library/AutoPkg/Cache/com.github.autopkg.pkg.Firefox_EN/Firefox/Applications
Copier
Copier: Mounted disk image /Users/tkimpton/Library/AutoPkg/Cache/com.github.autopkg.pkg.Firefox_EN/downloads/Firefox.dmg
Copier: Copied /private/tmp/dmg.8KiRNc/Firefox.app to /Users/tkimpton/Library/AutoPkg/Cache/com.github.autopkg.pkg.Firefox_EN/Firefox/Applications/Firefox.app
PkgCreator
PkgCreator: Connecting
PkgCreator: Disconnecting
Couldn't connect to autopkgserver: No such file or directory
Failed.
Receipt written to /Users/tkimpton/Library/AutoPkg/Cache/com.github.autopkg.pkg.Firefox_EN/receipts/Firefox-receipt-20140108-151600.plist

The following recipes failed:
Firefox.pkg
Error in com.github.autopkg.pkg.Firefox_EN: Processor: PkgCreator: Error: Couldn't connect to autopkgserver: No such file or directory

The following new items were downloaded:
/Users/tkimpton/Library/AutoPkg/Cache/com.github.autopkg.pkg.Firefox_EN/downloads/Firefox.dmg

To discuss how this should work. Here are my thoughts:

Assuming we could add a new recipe:

autopkg add-recipes https://github.com/foo/recipes.git

(using .git as a hint that it needs to git clone rather than unzip or mirror over a locally-accessible folder)

..it would git clone to a new location and update the RECIPE_SEARCH_DIRS preference array by adding the new location to the end of the list.

Currently we have ~/Library/AutoPkg/Recipes as a default recipe location, but I'm thinking it should be the default directory where recipes wind up as subdirectories with some namespacing.

From what little I've seen of Go, it manages third-party libraries separated by domain and path components. If your project imports "github.com/foo/recipes" as a dependency, it will clone this dependency into the src root that could be laid out like:

- github.com
  - foo
    - recipes
  - bar
    - MyAutoPkgRecipes
- internal.org.server
  - baz
    - corporate-recipes

autopkg could also provide the convenience of updating one or more recipe repos:

autopkg update-recipes --all

autopkg update-recipes autopkg

The second form would be simply a re.search match across the list of repo urls that we've cloned, so that I can just say "foo" to update anything that matches foo's recipes. Or, specify an exact URL.

Updating or recipe development could also be done with the git clones as one normally would, because they're just Git clones.

Currently, recipe overrides do not store the path to the original recipe; instead they use the search path (and search rules) to find the original recipe.

This is good because the override can find the original recipe even if it's moved to different directory, as long as it's in the search path.

This is bad, because if a git pull updates a recipe repo that's earlier in the search path and adds a new recipe of the same name as one lower in the search path, the override will now start using the recipe from a different repo, with unexpected results.

Recipes should be strongly encouraged to have unique identifiers. A Recipe override should store the recipe identifier of its original recipe, and use that identifier to find the original recipe.

Additionally, there are probably scenarios in which autopkg should check all recipes in its search paths for duplicate identifiers and warn. A good time to do this might be when adding or updating a recipe repo.