I'm seeing a problem with the JamfPolicyUploader processor when running various polici

It appears that it was. Please see below: <div class="snippet-clipboard-content no

JamfPolicyUploader processor failing with errors when updating policies,about grahampugh/jamf-upload

Comments (32)

grahampugh commented on August 11, 2024 1

This is basically resolved, but I'll leave the issue open until the Sleep processor is released, as a reminder to put that in the wiki once it's done. (Hint: if it's done before July 14 I can also mention the solution in our MacAdmins Campfire Talk....)

from jamf-upload.

grahampugh commented on August 11, 2024

Hi Rich. The error says "problem with package". That usually means the package that's been added to the policy doesn't exist. Can we look earlier in the run to see if the package and uploads were successful?

Edit: never mind, I now noticed the full output trace link!

So, it did upload successfully. I can only imagine that it's a timing issue. I notice that your trace does not recognise any cookies. I don't know if that's something specific to your set up, but it certainly means that each curl request is going to a random cluster node, which makes it entirely possible that node that's being hit to receive the policy is a different one to the one that received the package, and synchronisation hasn't happened yet.

All the cookie stuff is based on information I got regarding standard Jamf Cloud servers and the cookies they produce. I wonder if your server is different? We might need to analyse the curl headers.

from jamf-upload.

rtrouton commented on August 11, 2024

It appears that it was. Please see below:

JamfPackageUploader: No value supplied for pkg_name, setting default value of: 
JamfPackageUploader: No value supplied for pkg_info, setting default value of: 
JamfPackageUploader: No value supplied for reboot_required, setting default value of: 
JamfPackageUploader: No value supplied for os_requirements, setting default value of: 
JamfPackageUploader: No value supplied for required_processor, setting default value of: None
JamfPackageUploader: No value supplied for send_notification, setting default value of: 
JamfPackageUploader: No value supplied for replace_pkg, setting default value of: False
JamfPackageUploader: No value supplied for replace_pkg_metadata, setting default value of: False
JamfPackageUploader: No value supplied for SMB_URL, setting default value of: 
JamfPackageUploader: No value supplied for SMB_USERNAME, setting default value of: 
JamfPackageUploader: No value supplied for SMB_PASSWORD, setting default value of: 
JamfPackageUploader: Checking for existing 'Cisco_Umbrella_3.0.9.pkg' on https://jamf.pro.server.here
JamfPackageUploader: Checking for existing authentication token
JamfPackageUploader: Checking https://jamf.pro.server.here against https://jamf.pro.server.here
JamfPackageUploader: URL for token matches current request
JamfPackageUploader: Existing token is valid
JamfPackageUploader: No existing cookie found - starting new session
JamfPackageUploader: curl command: /usr/bin/curl --silent --show-error -D /tmp/jamf_upload/curl_headers_from_jamf_upload.txt --output /tmp/jamf_upload/curl_output_from_jamf_upload.txt https://jamf.pro.server.here/api/v1/jamf-pro-version --request GET --header authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJhdXRoZW50aWNhdGVkLWFwcCI6IkdFTkVSSUMiLCJhdXRoZW50aWNhdGlvbi10eXBlIjoiSlNTIiwiZ3JvdXBzIjpbXSwic3ViamVjdC10eXBlIjoiSlNTX1VTRVJfSUQiLCJ0b2tlbi11dWlkIjoiMzE3ZTcyMWQtMzBjYi00ZjM3LWE3MDEtMDhhN2QyMWJhYzM1IiwibGRhcC1zZXJ2ZXItaWQiOi0xLCJzdWIiOiIyMCIsImV4cCI6MTY0Njk0MDc2M30.g4Cv9b7TOPyZP1zWPNK1SMO5c-9LxnUGTMBGnuuyY0k --header Accept: application/json
JamfPackageUploader: Jamf Pro Version: 10.37.0-b.2.t1645716981
JamfPackageUploader: Token auth will be used, 
JamfPackageUploader: No existing cookie found - starting new session
JamfPackageUploader: curl command: /usr/bin/curl --silent --show-error -D /tmp/jamf_upload/curl_headers_from_jamf_upload.txt --output /tmp/jamf_upload/curl_output_from_jamf_upload.txt https://jamf.pro.server.here/JSSResource/packages/name/Cisco_Umbrella_3.0.9.pkg --request GET --header authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJhdXRoZW50aWNhdGVkLWFwcCI6IkdFTkVSSUMiLCJhdXRoZW50aWNhdGlvbi10eXBlIjoiSlNTIiwiZ3JvdXBzIjpbXSwic3ViamVjdC10eXBlIjoiSlNTX1VTRVJfSUQiLCJ0b2tlbi11dWlkIjoiMzE3ZTcyMWQtMzBjYi00ZjM3LWE3MDEtMDhhN2QyMWJhYzM1IiwibGRhcC1zZXJ2ZXItaWQiOi0xLCJzdWIiOiIyMCIsImV4cCI6MTY0Njk0MDc2M30.g4Cv9b7TOPyZP1zWPNK1SMO5c-9LxnUGTMBGnuuyY0k --header Accept: application/json
JamfPackageUploader: ID: -1
JamfPackageUploader: No existing cookie found - starting new session
JamfPackageUploader: curl command: /usr/bin/curl --silent --show-error -D /tmp/jamf_upload/curl_headers_from_jamf_upload.txt --output /tmp/jamf_upload/curl_output_from_jamf_upload.txt https://jamf.pro.server.here/dbfileupload --request POST --header authorization: Basic nopenothereZQ== --upload-file /Users/ec2-user/Library/AutoPkg/Cache/local.jamf.CiscoUmbrella-App-AutoUpdate/Cisco_Umbrella_3.0.9.pkg --header Accept: application/xml --header DESTINATION: 0 --header OBJECT_ID: -1 --header FILE_TYPE: 0 --header FILE_NAME: Cisco_Umbrella_3.0.9.pkg --max-time 3600
JamfPackageUploader: HTTP response: 200
JamfPackageUploader: Package uploaded successfully, ID=2371
JamfPackageUploader: Package 'Cisco_Umbrella_3.0.9.pkg' upload successful
JamfPackageUploader: Updating package metadata for 2371
JamfPackageUploader: <package><name>Cisco_Umbrella_3.0.9.pkg</name><filename>Cisco_Umbrella_3.0.9.pkg</filename><category>Apps</category><info></info><notes>Cisco Umbrella 3.0.9 installer uploaded via AutoPkg.</notes><priority>6</priority><reboot_required>False</reboot_required><required_processor>None</required_processor><os_requirements></os_requirements><hash_type>SHA_512</hash_type><hash_value>0627a2160d74b923e46bbc1a5c1cc1c5dc0b51a4b5171377841d3a3287186fc6a936c4a78eaeee237603a3fded8177619948959383833898cf5986480d29f183</hash_value><send_notification>False</send_notification></package>
JamfPackageUploader: Package metadata upload attempt 1
JamfPackageUploader: No existing cookie found - starting new session
JamfPackageUploader: curl command: /usr/bin/curl --silent --show-error -D /tmp/jamf_upload/curl_headers_from_jamf_upload.txt --output /tmp/jamf_upload/curl_output_from_jamf_upload.txt https://jamf.pro.server.here/JSSResource/packages/id/2371 --request PUT --header authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJhdXRoZW50aWNhdGVkLWFwcCI6IkdFTkVSSUMiLCJhdXRoZW50aWNhdGlvbi10eXBlIjoiSlNTIiwiZ3JvdXBzIjpbXSwic3ViamVjdC10eXBlIjoiSlNTX1VTRVJfSUQiLCJ0b2tlbi11dWlkIjoiMzE3ZTcyMWQtMzBjYi00ZjM3LWE3MDEtMDhhN2QyMWJhYzM1IiwibGRhcC1zZXJ2ZXItaWQiOi0xLCJzdWIiOiIyMCIsImV4cCI6MTY0Njk0MDc2M30.g4Cv9b7TOPyZP1zWPNK1SMO5c-9LxnUGTMBGnuuyY0k --upload-file /tmp/jamf_upload/jamf_upload_2658b3a6-899b-43e7-bff5-c97f313fd4d4.txt --header Content-type: application/xml
JamfPackageUploader: Package metadata 'Cisco_Umbrella_3.0.9.pkg' update successful
{'Output': {'jamfpackageuploader_summary_result': {'data': {'category': 'Apps',
                                                            'pkg_name': 'Cisco_Umbrella_3.0.9.pkg',
                                                            'pkg_path': '/Users/ec2-user/Library/AutoPkg/Cache/local.jamf.CiscoUmbrella-App-AutoUpdate/Cisco_Umbrella_3.0.9.pkg',
                                                            'version': '3.0.9'},
                                                   'report_fields': ['pkg_path',
                                                                     'pkg_name',
                                                                     'version',
                                                                     'category'],
                                                   'summary_text': 'The '
                                                                   'following '
                                                                   'packages '
                                                                   'were '
                                                                   'uploaded '
                                                                   'to or '
                                                                   'updated in '
                                                                   'Jamf Pro:'},
            'pkg_name': 'Cisco_Umbrella_3.0.9.pkg',
            'pkg_path': '/Users/ec2-user/Library/AutoPkg/Cache/local.jamf.CiscoUmbrella-App-AutoUpdate/Cisco_Umbrella_3.0.9.pkg',
            'pkg_uploaded': True}}
StopProcessingIf

from jamf-upload.

rtrouton commented on August 11, 2024

Assuming that is a timing issue, is there some way I can force a pause between running the JamfPackageUploader and JamfPolicyUploader processors? I’m on a Jamf Cloud Premium instance and definitely have multiple cluster nodes in play here.

from jamf-upload.

rtrouton commented on August 11, 2024

Ah, just thought of a way to force a step in between. I can move the category upload for the policy category from before the package upload to after the package upload.

from jamf-upload.

grahampugh commented on August 11, 2024

From what I understand it can take up to a minute for cluster sync to happen, so not working with cookies has a major impact. Sure, we could write a simple "sleep" processor but getting the correct cookies is the best way to solve this.

On the other hand, I don't remember you complaining about this with JSSImporter runs, which does the same thing with cookies, so I might be barking up the wrong tree. Let's see if the short amount of time spent processing the category is enough.

from jamf-upload.

rtrouton commented on August 11, 2024

As far as I can tell, this timing issue doesn’t happen with JSSImporter. I’m actually running Jamf Upload side by side with JSSImporter, where I have one AutoPkg instance with Jamf Upload feeding my test environment and a second AutoPkg instance with JSSImporter feeding my production environment, so I’ve been able to make that assessment between the two solutions.

from jamf-upload.

grahampugh commented on August 11, 2024

Could you post the contents of the file /tmp/jamf_upload/curl_cookies_from_jamf_upload.txt after a run?. Hopefully it will give some clues about the cookies. Also, running with -vvv shows cookie information for the package upload (not the other processes).

from jamf-upload.

rtrouton commented on August 11, 2024

The /tmp/jamf_upload/curl_cookies_from_jamf_upload.txt file doesn't appear to exist on my AutoPkg instance running Jamf Upload. I have the following files in /tmp/jamf_upload (see attached):

curl_headers_from_jamf_upload.txt
curl_output_from_jamf_upload.txt

curl_headers_from_jamf_upload.txt
curl_output_from_jamf_upload.txt

from jamf-upload.

rtrouton commented on August 11, 2024

I've updated my recipes to move the category upload for the policy category from before the package upload to after the package upload. We'll see if that makes a difference over the weekend.

from jamf-upload.

rtrouton commented on August 11, 2024

So far, it looks like moving the policy category upload to after the package upload has addressed the issue. I haven’t seen the issue re-appear after four days.

from jamf-upload.

rtrouton commented on August 11, 2024

Never mind, it just happened.

from jamf-upload.

rtrouton commented on August 11, 2024

Would it be possible to add a “wait” Input argument to the processor? Allow the value to be an arbitrary time in seconds and set the default to zero.

from jamf-upload.

rtrouton commented on August 11, 2024

That would allow folks with this problem to set their recipe to however many seconds are generally needed (in case the cookie isn’t there) for the package to register before trying to upload the policy.

from jamf-upload.

rtrouton commented on August 11, 2024

I noticed I hadn't upped the verbosity of logging on the AutoPkg instance where I saw the behavior. I've now done that and we'll see what happens.

Interestingly enough, I have not seen the policy update failure behavior on my test AutoPkg instance again, where I already had the logging verbosity turned up. Might be a coincidence, but something to add to the store of information about this issue.

from jamf-upload.

matthew-sec commented on August 11, 2024

I also have similar issues. I used to get the same issues on JSSImporter. I have also been occasionally getting them on the new JCDS upload method.

I was thinking that building a sleep processor might be simple enough, but I like the idea of building a wait into the processor as it is a simpler fix closer to the actual issue.

@rtrouton have you tested if a wait fixes the issue? My assumption is that it would but I haven't tested it.

from jamf-upload.

rtrouton commented on August 11, 2024

@matthew-sec,

The best way I could find to implement a "wait" was to move the category upload for the policy category from before the package upload to after the package upload. That may have improved matters somewhat by introducing a small delay at that point, but I'm still seeing the issue periodically.

from jamf-upload.

rtrouton commented on August 11, 2024

I saw the problem today on my production AutoPkg setup when Microsoft Edge updated.

I'm attaching verbose output from the run as JamfPolicyUploader_error_processor_20220317.txt and the following files:

curl_headers_from_jamf_upload.txt
curl_output_from_jamf_upload.txt

It looks like the package uploaded by this section of the verbose output:

com.github.grahampugh.jamf-upload.processors/JamfPackageUploader
{'Input': {'API_PASSWORD': 'api.password.here',
           'API_USERNAME': 'api.username.here',
           'JSS_URL': 'https://jamf.pro.server.here',
           'pkg_category': 'Apps',
           'pkg_notes': 'Microsoft Edge 99.0.1150.46 installer uploaded via '
                        'AutoPkg.',
           'pkg_path': '/Users/ec2-user/Library/AutoPkg/Cache/local.jamf.MicrosoftEdge-App-AutoUpdate/Microsoft_Edge_99.0.1150.46.pkg',
           'version': '99.0.1150.46'}}
JamfPackageUploader: No value supplied for pkg_name, setting default value of: 
JamfPackageUploader: No value supplied for pkg_info, setting default value of: 
JamfPackageUploader: No value supplied for pkg_priority, setting default value of: 10
JamfPackageUploader: No value supplied for reboot_required, setting default value of: 
JamfPackageUploader: No value supplied for os_requirements, setting default value of: 
JamfPackageUploader: No value supplied for required_processor, setting default value of: None
JamfPackageUploader: No value supplied for send_notification, setting default value of: 
JamfPackageUploader: No value supplied for replace_pkg, setting default value of: False
JamfPackageUploader: No value supplied for replace_pkg_metadata, setting default value of: False
JamfPackageUploader: No value supplied for SMB_URL, setting default value of: 
JamfPackageUploader: No value supplied for SMB_USERNAME, setting default value of: 
JamfPackageUploader: No value supplied for SMB_PASSWORD, setting default value of: 
JamfPackageUploader: Checking for existing 'Microsoft_Edge_99.0.1150.46.pkg' on https://jamf.pro.server.here
JamfPackageUploader: Checking for existing authentication token
JamfPackageUploader: Checking https://jamf.pro.server.here against https://jamf.pro.server.here
JamfPackageUploader: URL for token matches current request
JamfPackageUploader: Existing token is valid
JamfPackageUploader: No existing cookie found - starting new session
JamfPackageUploader: curl command: /usr/bin/curl --silent --show-error -D /tmp/jamf_upload/curl_headers_from_jamf_upload.txt --output /tmp/jamf_upload/curl_output_from_jamf_upload.txt https://jamf.pro.server.here/api/v1/jamf-pro-version --request GET --header authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJhdXRoZW50aWNhdGVkLWFwcCI6IkdFTkVSSUMiLCJhdXRoZW50aWNhdGlvbi10eXBlIjoiSlNTIiwiZ3JvdXBzIjpbXSwic3ViamVjdC10eXBlIjoiSlNTX1VTRVJfSUQiLCJ0b2tlbi11dWlkIjoiM2ZhMTdhNGItZjhkYy00NzQ4LWJkMzAtOTNiNDNmMDM3NWE4IiwibGRhcC1zZXJ2ZXItaWQiOi0xLCJzdWIiOiIyMCIsImV4cCI6MTY0NzU0OTE1Mn0.gqqkIDi4f6D1518GGqhZA9mZT94bMV2yj96sZqhuQHQ --header Accept: application/json
JamfPackageUploader: Jamf Pro Version: 10.36.0-t1644331466
JamfPackageUploader: Token auth will be used, 
JamfPackageUploader: No existing cookie found - starting new session
JamfPackageUploader: curl command: /usr/bin/curl --silent --show-error -D /tmp/jamf_upload/curl_headers_from_jamf_upload.txt --output /tmp/jamf_upload/curl_output_from_jamf_upload.txt https://jamf.pro.server.here/JSSResource/packages/name/Microsoft_Edge_99.0.1150.46.pkg --request GET --header authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJhdXRoZW50aWNhdGVkLWFwcCI6IkdFTkVSSUMiLCJhdXRoZW50aWNhdGlvbi10eXBlIjoiSlNTIiwiZ3JvdXBzIjpbXSwic3ViamVjdC10eXBlIjoiSlNTX1VTRVJfSUQiLCJ0b2tlbi11dWlkIjoiM2ZhMTdhNGItZjhkYy00NzQ4LWJkMzAtOTNiNDNmMDM3NWE4IiwibGRhcC1zZXJ2ZXItaWQiOi0xLCJzdWIiOiIyMCIsImV4cCI6MTY0NzU0OTE1Mn0.gqqkIDi4f6D1518GGqhZA9mZT94bMV2yj96sZqhuQHQ --header Accept: application/json
JamfPackageUploader: ID: -1
JamfPackageUploader: No existing cookie found - starting new session
JamfPackageUploader: curl command: /usr/bin/curl --silent --show-error -D /tmp/jamf_upload/curl_headers_from_jamf_upload.txt --output /tmp/jamf_upload/curl_output_from_jamf_upload.txt https://jamf.pro.server.here/dbfileupload --request POST --header authorization: Basic amFtZnByby1hdXRvcGtnOkY4aWFyS016Y1p6MlJ2VA== --upload-file /Users/ec2-user/Library/AutoPkg/Cache/local.jamf.MicrosoftEdge-App-AutoUpdate/Microsoft_Edge_99.0.1150.46.pkg --header Accept: application/xml --header DESTINATION: 0 --header OBJECT_ID: -1 --header FILE_TYPE: 0 --header FILE_NAME: Microsoft_Edge_99.0.1150.46.pkg --max-time 3600
JamfPackageUploader: HTTP response: 200
JamfPackageUploader: Package uploaded successfully, ID=2407
JamfPackageUploader: Package 'Microsoft_Edge_99.0.1150.46.pkg' upload successful
JamfPackageUploader: Updating package metadata for 2407
JamfPackageUploader: <package><name>Microsoft_Edge_99.0.1150.46.pkg</name><filename>Microsoft_Edge_99.0.1150.46.pkg</filename><category>Apps</category><info></info><notes>Microsoft Edge 99.0.1150.46 installer uploaded via AutoPkg.</notes><priority>10</priority><reboot_required>False</reboot_required><required_processor>None</required_processor><os_requirements></os_requirements><hash_type>SHA_512</hash_type><hash_value>604aab6c5876940cf3584d5dbb3cd99cef15934dc16921868bb92fcb6d15446d24937ee58840a4935cb24b0d5149cd532fb5dc89ed0a236e4afe32d895bbac09</hash_value><send_notification>False</send_notification></package>
JamfPackageUploader: Package metadata upload attempt 1
JamfPackageUploader: No existing cookie found - starting new session
JamfPackageUploader: curl command: /usr/bin/curl --silent --show-error -D /tmp/jamf_upload/curl_headers_from_jamf_upload.txt --output /tmp/jamf_upload/curl_output_from_jamf_upload.txt https://jamf.pro.server.here/JSSResource/packages/id/2407 --request PUT --header authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJhdXRoZW50aWNhdGVkLWFwcCI6IkdFTkVSSUMiLCJhdXRoZW50aWNhdGlvbi10eXBlIjoiSlNTIiwiZ3JvdXBzIjpbXSwic3ViamVjdC10eXBlIjoiSlNTX1VTRVJfSUQiLCJ0b2tlbi11dWlkIjoiM2ZhMTdhNGItZjhkYy00NzQ4LWJkMzAtOTNiNDNmMDM3NWE4IiwibGRhcC1zZXJ2ZXItaWQiOi0xLCJzdWIiOiIyMCIsImV4cCI6MTY0NzU0OTE1Mn0.gqqkIDi4f6D1518GGqhZA9mZT94bMV2yj96sZqhuQHQ --upload-file /tmp/jamf_upload/jamf_upload_c388d626-b091-4e1a-8062-d4b404a0ef78.txt --header Content-type: application/xml
JamfPackageUploader: Package metadata 'Microsoft_Edge_99.0.1150.46.pkg' update successful
{'Output': {'jamfpackageuploader_summary_result': {'data': {'category': 'Apps',
                                                            'pkg_name': 'Microsoft_Edge_99.0.1150.46.pkg',
                                                            'pkg_path': '/Users/ec2-user/Library/AutoPkg/Cache/local.jamf.MicrosoftEdge-App-AutoUpdate/Microsoft_Edge_99.0.1150.46.pkg',
                                                            'version': '99.0.1150.46'},
                                                   'report_fields': ['pkg_path',
                                                                     'pkg_name',
                                                                     'version',
                                                                     'category'],
                                                   'summary_text': 'The '
                                                                   'following '
                                                                   'packages '
                                                                   'were '
                                                                   'uploaded '
                                                                   'to or '
                                                                   'updated in '
                                                                   'Jamf Pro:'},
            'pkg_name': 'Microsoft_Edge_99.0.1150.46.pkg',
            'pkg_path': '/Users/ec2-user/Library/AutoPkg/Cache/local.jamf.MicrosoftEdge-App-AutoUpdate/Microsoft_Edge_99.0.1150.46.pkg',
            'pkg_uploaded': True}}
StopProcessingIf
{'Input': {'predicate': 'pkg_uploaded == FALSE'}}
StopProcessingIf: (pkg_uploaded == FALSE) is False

However the policy update fails when the package can't be found (presumably because now Jamf Upload is talking to a new cluster node.):

com.github.grahampugh.jamf-upload.processors/JamfPolicyUploader
{'Input': {'API_PASSWORD': 'api.password.here',
           'API_USERNAME': 'api.username.here',
           'JSS_URL': 'https://jamf.pro.server.here',
           'policy_name': 'Auto Update MicrosoftEdge',
           'policy_template': 'PolicyTemplate-autoupdate.xml',
           'replace_policy': 'True'}}
JamfPolicyUploader: No value supplied for icon, setting default value of: 
JamfPolicyUploader: No value supplied for replace_icon, setting default value of: False
JamfPolicyUploader: File found at: /Users/ec2-user/Library/AutoPkg/RecipeRepos/com.github.rtrouton.autopkg_recipes/Jamf_Upload/PolicyTemplate-autoupdate.xml
JamfPolicyUploader: ("Replacing any instances of 'POLICY_NAME' with", "'Auto Update MicrosoftEdge'")
JamfPolicyUploader: ("Replacing any instances of 'POLICY_CATEGORY' with", "'Apps-AutoUpdate'")
JamfPolicyUploader: ("Replacing any instances of 'POLICY_TRIGGER' with", "'autoupdate-MicrosoftEdge'")
JamfPolicyUploader: ("Replacing any instances of 'pkg_name' with", "'Microsoft_Edge_99.0.1150.46.pkg'")
JamfPolicyUploader: Policy data:
JamfPolicyUploader: <?xml version="1.0" encoding="UTF-8"?>
<policy>
   <general>
      <name>Auto Update MicrosoftEdge</name>
      <enabled>true</enabled>
      <frequency>Ongoing</frequency>
      <category>
         <name>Apps-AutoUpdate</name>
      </category>
      <trigger_other>autoupdate-MicrosoftEdge</trigger_other>
   </general>
   <scope>
      <all_computers>true</all_computers>
   </scope>
   <package_configuration>
      <packages>
         <size>1</size>
         <package>
            <name>Microsoft_Edge_99.0.1150.46.pkg</name>
            <action>Install</action>
         </package>
      </packages>
   </package_configuration>
   <scripts>
      <size>0</size>
   </scripts>
   <maintenance>
      <recon>true</recon>
   </maintenance>
</policy>
JamfPolicyUploader: Checking for existing 'Auto Update MicrosoftEdge' on https://jamf.pro.server.here
JamfPolicyUploader: Checking for existing authentication token
JamfPolicyUploader: Checking https://jamf.pro.server.here against https://jamf.pro.server.here
JamfPolicyUploader: URL for token matches current request
JamfPolicyUploader: Existing token is valid
JamfPolicyUploader: No existing cookie found - starting new session
JamfPolicyUploader: curl command: /usr/bin/curl --silent --show-error -D /tmp/jamf_upload/curl_headers_from_jamf_upload.txt --output /tmp/jamf_upload/curl_output_from_jamf_upload.txt https://jamf.pro.server.here/api/v1/jamf-pro-version --request GET --header authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJhdXRoZW50aWNhdGVkLWFwcCI6IkdFTkVSSUMiLCJhdXRoZW50aWNhdGlvbi10eXBlIjoiSlNTIiwiZ3JvdXBzIjpbXSwic3ViamVjdC10eXBlIjoiSlNTX1VTRVJfSUQiLCJ0b2tlbi11dWlkIjoiM2ZhMTdhNGItZjhkYy00NzQ4LWJkMzAtOTNiNDNmMDM3NWE4IiwibGRhcC1zZXJ2ZXItaWQiOi0xLCJzdWIiOiIyMCIsImV4cCI6MTY0NzU0OTE1Mn0.gqqkIDi4f6D1518GGqhZA9mZT94bMV2yj96sZqhuQHQ --header Accept: application/json
JamfPolicyUploader: Jamf Pro Version: 10.36.0-t1644331466
JamfPolicyUploader: Token auth will be used, 
JamfPolicyUploader: No existing cookie found - starting new session
JamfPolicyUploader: curl command: /usr/bin/curl --silent --show-error -D /tmp/jamf_upload/curl_headers_from_jamf_upload.txt --output /tmp/jamf_upload/curl_output_from_jamf_upload.txt https://jamf.pro.server.here/JSSResource/policies --request GET --header authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJhdXRoZW50aWNhdGVkLWFwcCI6IkdFTkVSSUMiLCJhdXRoZW50aWNhdGlvbi10eXBlIjoiSlNTIiwiZ3JvdXBzIjpbXSwic3ViamVjdC10eXBlIjoiSlNTX1VTRVJfSUQiLCJ0b2tlbi11dWlkIjoiM2ZhMTdhNGItZjhkYy00NzQ4LWJkMzAtOTNiNDNmMDM3NWE4IiwibGRhcC1zZXJ2ZXItaWQiOi0xLCJzdWIiOiIyMCIsImV4cCI6MTY0NzU0OTE1Mn0.gqqkIDi4f6D1518GGqhZA9mZT94bMV2yj96sZqhuQHQ --header Accept: application/json
JamfPolicyUploader: Policy 'Auto Update MicrosoftEdge' already exists: ID 1664
JamfPolicyUploader: Replacing existing policy as 'replace_policy' is set to True
JamfPolicyUploader: Uploading Policy...
JamfPolicyUploader: Policy upload attempt 1
JamfPolicyUploader: No existing cookie found - starting new session
JamfPolicyUploader: curl command: /usr/bin/curl --silent --show-error -D /tmp/jamf_upload/curl_headers_from_jamf_upload.txt --output /tmp/jamf_upload/curl_output_from_jamf_upload.txt https://jamf.pro.server.here/JSSResource/policies/id/1664 --request PUT --header authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJhdXRoZW50aWNhdGVkLWFwcCI6IkdFTkVSSUMiLCJhdXRoZW50aWNhdGlvbi10eXBlIjoiSlNTIiwiZ3JvdXBzIjpbXSwic3ViamVjdC10eXBlIjoiSlNTX1VTRVJfSUQiLCJ0b2tlbi11dWlkIjoiM2ZhMTdhNGItZjhkYy00NzQ4LWJkMzAtOTNiNDNmMDM3NWE4IiwibGRhcC1zZXJ2ZXItaWQiOi0xLCJzdWIiOiIyMCIsImV4cCI6MTY0NzU0OTE1Mn0.gqqkIDi4f6D1518GGqhZA9mZT94bMV2yj96sZqhuQHQ --upload-file /tmp/jamf_upload/jamf_upload_633d3dc9-7dc5-43a5-9300-4481252f5cd4.txt --header Content-type: application/xml
JamfPolicyUploader: b'<html>\n<head>\n   <title>Status page</title>\n</head>\n<body style="font-family: sans-serif;">\n<p style="font-size: 1.2em;font-weight: bold;margin: 1em 0px;">Conflict</p>\n<p>Error: Problem with package</p>\n<p>You can get technical details <a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.10">here</a>.<br>\nPlease continue your visit at our <a href="/">home page</a>.\n</p>\n</body>\n</html>\n'
Traceback (most recent call last):
  File "/Library/AutoPkg/autopkglib/__init__.py", line 840, in process
    self.env = processor.process()
  File "/Library/AutoPkg/autopkglib/__init__.py", line 626, in process
    self.main()
  File "/Users/ec2-user/Library/AutoPkg/RecipeRepos/com.github.autopkg.grahampugh-recipes/JamfUploaderProcessors/JamfPolicyUploader.py", line 304, in main
    token=token,
  File "/Users/ec2-user/Library/AutoPkg/RecipeRepos/com.github.autopkg.grahampugh-recipes/JamfUploaderProcessors/JamfPolicyUploader.py", line 134, in upload_policy
    if self.status_check(r, "Policy", policy_name, request) == "break":
  File "/Users/ec2-user/Library/AutoPkg/RecipeRepos/com.github.autopkg.grahampugh-recipes/JamfUploaderProcessors/JamfUploaderLib/JamfUploaderBase.py", line 396, in status_check
    f"WARNING: {endpoint_type} '{obj_name}' {action} failed due to a conflict"
autopkglib.ProcessorError: WARNING: Policy 'Auto Update MicrosoftEdge' update failed due to a conflict
  File "/Library/AutoPkg/autopkglib/__init__.py", line 840, in process
    self.env = processor.process()
Receipt written to /Users/ec2-user/Library/AutoPkg/Cache/local.jamf.MicrosoftEdge-App-AutoUpdate/receipts/local.jamf-receipt-20220317-201330.plist

curl_headers_from_jamf_upload.txt
curl_output_from_jamf_upload.txt

JamfPolicyUploader_error_processor_20220317.txt

If the policy update hits a failure, could the JamfPolicyUploader processor wait a certain amount of time (say 30 seconds) and then try a second time?

from jamf-upload.

grahampugh commented on August 11, 2024

Just to let you know, I'm on vacation for a fe more days. I'll look into this next week. the headers file looks helpful, at least some cookies are being generated so I can work with this.

I think the solution is getting the cookies to work rather than building workarounds like "sleep" processors. Jamf should be able to help with this.

from jamf-upload.

rtrouton commented on August 11, 2024

I have a ticket open with Jamf and I've pointed them at this GitHub issue report, as well as mentioning the cookie issue.

from jamf-upload.

grahampugh commented on August 11, 2024

@rtrouton just taking another look at this and specifically the repeated No existing cookie found - starting new session messages. The only circumstance that should generate that output is if the cookie_jar file does not exist. The cookie jar file is written for any curl request to any jamf API endpoint (api, uapi, JSSResource, dbfileupload, legacy/packages).

Can you check if no such file is being written to /tmp/jamf_upload/curl_cookies_from_jamf_upload.txt, and if not, could there be something on your system preventing this being written?

from jamf-upload.

rtrouton commented on August 11, 2024

I'm seeing the following files in /tmp/jamf_upload:

curl_cookies_from_jamf_upload.txt
curl_headers_from_jamf_upload.txt
curl_output_from_jamf_upload.txt

Please see below for output:

username@computername ~ % cd /tmp/jamf_upload
username@computername jamf_upload % ls -al
total 24
drwxr-xr-x    5 username  wheel   160 May 10 06:21 .
drwxrwxrwt  114 root      wheel  3648 May 10 13:35 ..
-rw-r--r--    1 username  wheel   509 May 10 13:24 curl_cookies_from_jamf_upload.txt
-rw-r--r--    1 username  wheel   703 May 10 13:24 curl_headers_from_jamf_upload.txt
-rw-r--r--    1 username  wheel   105 May 10 13:24 curl_output_from_jamf_upload.txt
username@computername jamf_upload %

from jamf-upload.

rtrouton commented on August 11, 2024

When I checked the contents of curl_cookies_from_jamf_upload.txt, here's what I saw (Jamf Pro server's DNS address removed from output):

username@computername jamf_upload % cat curl_cookies_from_jamf_upload.txt
# Netscape HTTP Cookie File
# https://curl.haxx.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.

jamf.pro.server.address.here	FALSE	/	TRUE	1652793886	AWSALBCORS	opiems8RhKYXk5HYMg4DpWe+xp0mFuMz09YLOQwtf6MaYWT9gt5YKIgemqB/HkrIbTDCNkCAnNXkNcwpsgvlDL1j1pXCUi02vRwUbd2SKQEPdWYUghsusSb7FOb3
jamf.pro.server.address.here	FALSE	/	FALSE	1652793886	AWSALB	opiems8RhKYXk5HYMg4DpWe+xp0mFuMz09YLOQwtf6MaYWT9gt5YKIgemqB/HkrIbTDCNkCAnNXkNcwpsgvlDL1j1pXCUi02vRwUbd2SKQEPdWYUghsusSb7FOb3
username@computername jamf_upload %

from jamf-upload.

grahampugh commented on August 11, 2024

Hmm this is very strange. I have now noticed in your output that no cookie is being set. There should be --cookie-jar /tmp/jamf_upload/curl_cookies_from_jamf_upload.txt and --cookie /tmp/jamf_upload/curl_cookies_from_jamf_upload.txt in each curl output.

Would you please run a verbose trace again and see if this is still the case? I think I have made changes since you ran this in March which might have fixed it.

from jamf-upload.

rtrouton commented on August 11, 2024

Sure thing, I'll give that a try tomorrow.

from jamf-upload.

rtrouton commented on August 11, 2024

It looks like the curl command now includes the --cookie-jar and --cookie values:

JamfPackageUploader: curl command: /usr/bin/curl --dump-header /tmp/jamf_upload/curl_headers_from_jamf_upload.txt https://jamf.pro.server.goes.here/api/v1/jamf-pro-version --request GET --silent --show-error --header authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJhdXRoZW50aWNhdGVkLWFwcCI6IkdFTkVSSUMiLCJhdXRoZW50aWNhdGlvbi10eXBlIjoiSlNTIiwiZ3JvdXBzIjpbXSwic3ViamVjdC10eXBlIjoiSlNTX1VTRVJfSUQiLCJ0b2tlbi11dWlkIjoiYTM1NjZkMDUtZjMzOC00OTk4LTkyYWEtMGE0MmY2YmI4ZTYyIiwibGRhcC1zZXJ2ZXItaWQiOi0xLCJzdWIiOiIyMCIsImV4cCI6MTY1MjQ0ODgxMn0.rpv7DPGARSdH0HnmN-qTdH2k-ysGI1bvSWT-hUYLhjE --output /tmp/jamf_upload/curl_output_from_jamf_upload.txt --output /tmp/jamf_upload/curl_output_from_jamf_upload.txt --header Accept: application/json --cookie-jar /tmp/jamf_upload/curl_cookies_from_jamf_upload.txt --cookie /tmp/jamf_upload/curl_cookies_from_jamf_upload.txt

I'm attaching the full output as a separate file (please see attached "autopkg_verbose_output.txt" file.)

Thanks,
Rich

autopkg_verbose_output.txt

from jamf-upload.

grahampugh commented on August 11, 2024

Hi @rtrouton
Thanks for the output. It looks like the cookies are working as intended now. I've also had confirmation from Jamf that the processors "do the right thing" in terms of use of the cookie jar. They did also say that the unofficial endpoints might not respect the cookies, especially not with third-party fileshare repos, which if the case could explain the problems you have been experiencing.

I see you have added a "sleep" processor, which is presumably solving your problem. Given the above information, it seems a sensible workaround if you otherwise have problems. I would be, however, interested to see if it's still necessary.

But no stress on that if it's working for you with the sleep step.

from jamf-upload.

grahampugh commented on August 11, 2024

FYI, I'm integrating sleep into JamfUploader processors. I've tested with JamfComputerGroupUploader and it's working well for me - I needed to add a 2-second sleep since upgrading to Jamf Pro 10.39.1 to prevent failures. I guess it will also need to be added to some of the other processors that may contain dependencies - Policy, Patch and Profile spring to mind. I'll release this soon, but this is just to let you know that a Sleep processor will be probably superfluous (although I released one anyway while testing this stuff out).

from jamf-upload.

rtrouton commented on August 11, 2024

Thank you!

from jamf-upload.

bwadeIDS commented on August 11, 2024

Hi All, I am having some issues with JamfUploader since I installed and configured it yesterday and was hoping for some advice.

My apologies I don't have a lot of (if any!) experience in Mac scripting, I was tasked with managing our companies Macs with very little knowledge and found Autopkg, AutoPkgr and JSS Importer. I don't create my own recipes but have used other JSS recipes and have found these worked well for us. I saw that JSS was being deprecated so I went and configured JamfUploader.

A bit of background on my setup, I have a Jamf Pro Cloud instance, and use AutoPkgr with AutoPkg, JSSImporter and now JamfUploader.
I use the same Jamf login details for JSS Importer for Jamf Uploader.
I configured JamfUploader with my AutoPkgr instance yesterday and found that only a few of the recipes are working for me.

GoogleChrome.Jamf successfully uploaded a package, created a smart group, created a category and then finally made a policy.

Firefox.Jamf however will not, and only uploads the package file, it doesn't create a category, smart group or policy.

I have the same behaviour with the other following recipes:

DropBox.Jamf
GotoMeeting.Jamf
MicrosoftAutoUpdate.Jamf
Miro.Jamf
Skype.Jamf
Spotify.Jamf
VLC.Jamf
zoom.us.jamf

I created an override for each of these recipes and still have the same issue.

As a further piece of information which might be useful, the Firefox recipe seemed to progress further when I manually created the category it was trying to create. The recipe then progressed and created a smart group, but still did not create a policy. I am afraid I don't have the exact error message to hand but I recall the error after running this recipe stated something along the lines of 'unable to use substitute key' -(apologies for the vagueness!). This I believe was when it was trying to create the policy.

.JSS files are still uploading without issue however my MS Teams notification integration is posting blank messages after it runs since updating autoPkgr though that is a separate issue, and not as important.

I've attached the log of a Google Chrome.jamf run and 2 runs where they only a package file was uploaded to my Jamf Pro and no other steps were taken.

MiroJamf.txt
GoogleChromeJamf.txt
FireFoxJamf.txt

I'm struggling to troubleshoot so my apologies for the lack of other technical information that may be required. If there is any advice you able to offer it would be greatly appreciated!

Ben

from jamf-upload.

grahampugh commented on August 11, 2024

Hi Ben, it would be better if you start a new issue as your issue does not relate to this one.

Basically it looks like some of my example recipes don't have the JamfCategoryUploader processor in them, so they can't create a category that doesn't exist. I should have added that processor but overlooked it.

Current workaround is to create the category manually, or override the category with one you already have.

from jamf-upload.

rtrouton commented on August 11, 2024

My issue appears solved, so I'm closing this issue.

from jamf-upload.

JamfPolicyUploader processor failing with errors when updating policies about jamf-upload HOT 32 CLOSED

Comments (32)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent