valya: This can be easily accomplished by using HTTP headers. Here is discussion with Simon.

Hi,
I think taking the HTTP header is the best general solution. I guess you could co-routine something to over ride that if you find a DAS header...

This does bring up the question - do we still need the DAS header? What do you use it for these days? There are services which don't have it, so is it really needed?
Cheers
Simon

On 4 Oct 2010, at 20:37, Kuznetsov Valentin wrote:

Hi Simon,
while working with DAS header I fall into one problem. The code I have reads data from data-service in a lazy way. I do use generators. So I open URL and parse it as I go through. This proven to be very efficient (in memory consumption way) for parsing large XML outputs. I apply the same for JSON. So, when I address the issue with reading expire part from DAS compliant data-service I fall into the problem. Due to generator nature I can't read the header, until the time I start looping over the data. This step is only happens by final routine which inject data into mongo. At this point I already strip off everything (including DAS header info, since it is not part of records themselves). Imaging

rec = {'response_expires': 123, 'results': [{record1}, {record2}] }

So, the expire timestamp (response_expires) is not part of the records structure. In a code I look at record content, so I ask for rec['results'] in order to do the processing and yield the records.

But at the same time I want to read the DAS header since it contains expire timestamp. One easy way to deal with that would be to read 'Expires' from HTTP header, rather the data itself. I had a look at WebTools/Page.py code and do see that we "suppose" to set Expires in HTTP header. Do you think of any problems if I'll read Expires from HTTP header instead of DAS record?

Basically I will do this

{{{
data = urllib2.urlopen(req) # ask for data from data-service
info = data.info() # it contains HTTP header info and therefore Expires
...

code which deal with data, where data is file-like object I read, for instance

res = json.load(data)
yield res

process res

for row in res:
# process row

insert rows into DB

...
}}}

I looked at Tie0data-service, and able to read Expires in this way, but I found that Expires in HTTP header is not the same as Expires in das header of the returned records. May be it is a bug. But according to WebTools/Page.py

{{{
def exposedasjson (func):
def wrapper (self, data, expires, contentType = "application/json"):
data = runDas(self, func, data, expires)
try:
jsondata = JsonWrapper.dumps(data)
_setCherryPyHeaders(jsondata, contentType, expires)
return jsondata
except:
raise
.....

def runDas(self, func, data, expires):
"""
Run a query and produce a dictionary for DAS formatting
"""
start_time = time.time()
results = func(self, data)
call_time = time.time() - start_time
res_expire = make_timestamp(expires)

...
dasdata = {'application':'%s.%s' % (self.config.application, func.name),
'request_timestamp': start_time,
'request_url': request.base + request.path_info + '?' +
request.query_string,
'request_method' : request.method,
'request_params' : request.params,
'response_version': res_version,
'response_expires': res_expire,
'response_checksum': res_checksum,
'request_call': func.name,
'call_time': call_time,
'results': results,
}
return dasdata
}}}

Please let me know what do you think.
V.

from das.

valya: (In 29f1b00) Work on code based on Code Audit, fixes #290, #493, #494, #446, #449, #447

Signed-off-by: Valentin Kuznetsov [email protected]

from das.