[Open.ogc] Deleted layers still exist?
Micah Wengren
micah.wengren at noaa.gov
Tue Sep 24 19:32:19 UTC 2013
John,
You may have hit on an issue with the way our system works. Can you
tell if the attribute values that are being saved in the database are
coming from the new shapefile you just tried to upload? Could you
insert a test value and try the upload again? I just want to check that
the new data is actually being written when you specify
update=overwrite. It should be, but it would be good to confirm.
I don't actually know how GeoServer handles the overwrite option behind
the scenes. I thought it might drop the underlying database table and
recreate, but it may not be. That would be why the schema is being
preserved. Best way to find out is for me to open a ticket with
OpenGeo. I'll do that later today.
Micah
On 9/24/2013 2:59 PM, John Kennedy - NOAA Affiliate wrote:
> Okay, I now feel a little silly. I did not replace the dummy password
> in my notes file with my LDAP password. Shape files and zip files have
> the same names as the layer. I attempted the data load again, using
> --update overwrite, and I got a 201 response. However, when I look at
> the layer in geoserver I see the old data. There is a column
> (property) called GEOID_jlf and that column is in the old data layer
> and not in the new layer I'm attempting to load.
>
> Inline image 1
>
>
> John F. Kennedy, PhD, CPG
> GIS Specialist
> Contractor: Caelum Corp.
> NOAA Fisheries Office of Science and Technology
> 1315 East-West Highway
> SSMC3 Rm 9518
> Silver Spring, MD 20910
> 301-427-8149
>
> Comments? Have I been of assistance? Please let my supervisors know. :~)
> Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov
> <mailto:Tim.Haverland at noaa.gov>>
> Hussain Jabalpurwala - NOAA Affiliate <hussain.jabalpurwala at noaa.gov
> <mailto:hussain.jabalpurwala at noaa.gov>>
>
>
> On Tue, Sep 24, 2013 at 2:35 PM, Micah Wengren <micah.wengren at noaa.gov
> <mailto:micah.wengren at noaa.gov>> wrote:
>
> Ok. Just curious, what are the shapefile names you're trying to
> upload? Are they the communities_pts and communities_pgs layers?
>
> Looks like those are gone from the FeatureTypes list now
> (https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis.html).
> But they still show up in the list when you go to the 'New Layer'
> page. Not sure I've seen that issue before, so might have to file
> a ticket.
>
> As for credentials, it should only be your NOAA LDAP username and
> password. Those don't get passed on to the database, there's a
> separate way database authorization is handled. If you can log in
> to the Web UI, there shouldn't be an issue with the REST API, or
> at least you shouldn't be getting that 401 response.
>
> Micah
>
>
> On 9/24/2013 2:21 PM, John Kennedy - NOAA Affiliate wrote:
>> Thanks. Tomorrow works better for me. As for the credentials,
>> nothing as changed on this end. I was able to load data with the
>> hpcc_data.py script up to yesterday, but I did get an error
>> message when attempting to overwrite the layer/feature type. Now,
>> there maybe credential issues in the database (PostGIS?) but I've
>> no way of knowing.
>>
>> John F. Kennedy, PhD, CPG
>> GIS Specialist
>> Contractor: Caelum Corp.
>> NOAA Fisheries Office of Science and Technology
>> 1315 East-West Highway
>> SSMC3 Rm 9518
>> Silver Spring, MD 20910
>> 301-427-8149 <tel:301-427-8149>
>>
>> Comments? Have I been of assistance? Please let my supervisors
>> know. :~)
>> Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov
>> <mailto:Tim.Haverland at noaa.gov>>
>> Hussain Jabalpurwala - NOAA Affiliate
>> <hussain.jabalpurwala at noaa.gov
>> <mailto:hussain.jabalpurwala at noaa.gov>>
>>
>>
>> On Tue, Sep 24, 2013 at 2:15 PM, Micah Wengren
>> <micah.wengren at noaa.gov <mailto:micah.wengren at noaa.gov>> wrote:
>>
>> Hi John,
>>
>> That is odd for the 401 response on the data upload. Are you
>> sure the credentials are right? The other two seem ok,
>> depending on if the featuretype existed in the first place or
>> not.
>>
>> I might be able to come by later, but right now I have a
>> deadline to finish something this afternoon. Let me know if
>> you aren't able to get it to work.
>>
>> Micah
>>
>>
>> On 9/24/2013 12:32 PM, John Kennedy - NOAA Affiliate wrote:
>>> Hi Micah,
>>>
>>> Well, I think I'm getting close. I installed pip, python
>>> setup, and request. I used hpcc_date_requests.py to delete
>>> the layer and the feature type and I get these to responses:
>>> Response: 200,
>>> url=https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis/featuretypes/communities_pgs
>>>
>>> Response: 404,
>>> url=https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis/featuretypes/communities_pgs
>>>
>>> The hpcc_date_requests appears to be attempting to load
>>> data, but the response back is this:
>>> Response: 401,
>>> url=https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis/file.shp?update=overwrite&configure=all
>>>
>>> Any chance you can come to my cube and have a look?
>>>
>>> Cheers,
>>> John
>>>
>>>
>>>
>>> John F. Kennedy, PhD, CPG
>>> GIS Specialist
>>> Contractor: Caelum Corp.
>>> NOAA Fisheries Office of Science and Technology
>>> 1315 East-West Highway
>>> SSMC3 Rm 9518
>>> Silver Spring, MD 20910
>>> 301-427-8149 <tel:301-427-8149>
>>>
>>> Comments? Have I been of assistance? Please let my
>>> supervisors know. :~)
>>> Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov
>>> <mailto:Tim.Haverland at noaa.gov>>
>>> Hussain Jabalpurwala - NOAA Affiliate
>>> <hussain.jabalpurwala at noaa.gov
>>> <mailto:hussain.jabalpurwala at noaa.gov>>
>>>
>>>
>>> On Tue, Sep 24, 2013 at 10:18 AM, Micah Wengren
>>> <micah.wengren at noaa.gov <mailto:micah.wengren at noaa.gov>> wrote:
>>>
>>> Hi John,
>>>
>>> How are you doing the attempted data upload? With the
>>> python script? Can you tell me which python script you
>>> are using if so? If you can, try using the
>>> hpcc_data_requests.py script instead of the other one
>>> (might involve some admin rights to install python
>>> modules on your workstation). I've update that one more
>>> recently. I'd like to drop the other one at some point
>>> for easier maintenance.
>>>
>>> I think you should be able to upload a new data set to
>>> replace the existing one, even if it already exists in
>>> the database and data store (but could be wrong if
>>> there's an issue going on here with these layers).
>>> Something like the following should work:
>>>
>>> > hpcc_data_requests.py -a data -u <name> -p <pass> -w
>>> nmfs_st -f /path/to/shapefile.zip --type shape --update
>>> overwrite (the shapefile would need to be named to
>>> match the layer you want to update).
>>>
>>>
>>> Also, this is something that is a little whacky with the
>>> GeoServer REST API (and should be better documented
>>> somewhere admittedly), but you may need to do a delete
>>> command for both the Layer and the FeatureType to fully
>>> delete it. When you create a new layer with the Web UI,
>>> it actually makes both the Layer and FeatureType behind
>>> the scenes. Not sure if this will fix this particular
>>> issue, but commands to do that would be something like:
>>>
>>> > hpcc_data_requests.py -a delete_layer -u <name> -p
>>> <pass> -w nmfs_st --layername <layername>
>>> > hpcc_data_requests.py -a delete_featuretype -u <name>
>>> -p <pass> -w nmfs_st --layername <layername>
>>>
>>> This should fully delete both a layer and its
>>> corresponding featuretype. It's possible if you did a
>>> delete_layer command with via the script without doing
>>> delete_featuretype as well, some featuretypes got left
>>> behind. Again, an issue that should really be better
>>> documented in GeoServer docs but unfortunately I haven't
>>> seen it described anywhere, I happened to find out
>>> myself after having some similar issues.
>>>
>>> I'm looking around the workspace now, and I think I see
>>> some cases where there are FeatureTypes that aren't
>>> associated with any layers, in particular
>>> 'communities_pgs' and 'communities_pts' are both
>>> FeatureTypes without corresponding layers. This could
>>> be preventing the data upload from happening for these
>>> layers. I'm just looking at these two lists:
>>>
>>> https://services.ogc.noaa.gov/geoserver/web/?wicket:bookmarkablePage=:org.geoserver.web.data.layer.LayerPage
>>> (filtered by 'nmfs_st')
>>> https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis.html
>>>
>>> Also, you should probably do the delete_featuretype
>>> command for 'communities_pgs20130506' and
>>> 'communities_pts20130506', since they don't have
>>> corresponding Layers. I think that will get those two
>>> in sync.
>>>
>>> I'm not sure why you get such a long list in the New
>>> Layer dialog though, looks like a bug to me. No idea
>>> where 'SI_SE_Gulf_sites_pg_ALLsites' and
>>> 'SI_SE_Gulf_sites_pt' are coming from, because those
>>> aren't listed as current FeatureTypes in the REST API.
>>>
>>> HTH,
>>> Micah
>>>
>>>
>>>
>>>
>>>
>>> On 9/24/2013 9:30 AM, John Kennedy - NOAA Affiliate wrote:
>>>> The two layers of concern are communities_pgs
>>>> and communities_pts. When I attempt to load new data,
>>>> via a zip file with a shape file, I get a message that
>>>> the communities_pgs exists. When I look at the list of
>>>> source under the "Add sources" button I can see
>>>> the communities_pgs layer and the publish link. But
>>>> when I click the link and attempt to save I get the
>>>> message below. Deleting layer via the hpcc_data python
>>>> script also seems to fail.
>>>>
>>>> Inline image 1
>>>>
>>>> John F. Kennedy, PhD, CPG
>>>> GIS Specialist
>>>> Contractor: Caelum Corp.
>>>> NOAA Fisheries Office of Science and Technology
>>>> 1315 East-West Highway
>>>> SSMC3 Rm 9518
>>>> Silver Spring, MD 20910
>>>> 301-427-8149 <tel:301-427-8149>
>>>>
>>>> Comments? Have I been of assistance? Please let my
>>>> supervisors know. :~)
>>>> Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov
>>>> <mailto:Tim.Haverland at noaa.gov>>
>>>> Hussain Jabalpurwala - NOAA Affiliate
>>>> <hussain.jabalpurwala at noaa.gov
>>>> <mailto:hussain.jabalpurwala at noaa.gov>>
>>>>
>>>>
>>>> On Tue, Sep 24, 2013 at 9:20 AM, Tim Haverland - NOAA
>>>> Federal <tim.haverland at noaa.gov
>>>> <mailto:tim.haverland at noaa.gov>> wrote:
>>>>
>>>> I'm adding John Kennedy to this thread so he can
>>>> work with you on this issue.
>>>>
>>>> Tim
>>>>
>>>>
>>>> On Tue, Sep 24, 2013 at 8:49 AM, Micah Wengren
>>>> <micah.wengren at noaa.gov
>>>> <mailto:micah.wengren at noaa.gov>> wrote:
>>>>
>>>> Tim,
>>>>
>>>> I can see some layers in your workspace in the
>>>> New Layer dialog that aren't currently
>>>> published and have a 'Publish' link next to
>>>> them, so I see what you mean. Are you able to
>>>> create a new SQL view layer with the same name
>>>> as one of those layers, or does that not work?
>>>> Can you give me the name of one of the layers
>>>> that you've tried with and I'll pass it on to
>>>> OpenGeo support to troubleshoot, that way I can
>>>> give them a specific layer name with the issue.
>>>>
>>>> The layers should be disappearing but for
>>>> whatever reason they're not.
>>>>
>>>> Micah
>>>>
>>>>
>>>>
>>>> On 9/23/2013 3:55 PM, Tim Haverland - NOAA
>>>> Federal wrote:
>>>>> Hi all,
>>>>>
>>>>> We've been trying to clean up a bunch of
>>>>> layers in our workspace. We can delete layers
>>>>> using the "Remove selected resources" button
>>>>> on the Layers page, but when we try to replace
>>>>> that layer with another, the layer is still
>>>>> there.
>>>>>
>>>>> We can verify this by clicking "Add a new
>>>>> resource" ... all of the old layers are still
>>>>> there and I can click Publish to make them
>>>>> "live" again.
>>>>>
>>>>> How do we permanently get rid of unwanted layers?
>>>>>
>>>>> Tim
>>>>> --
>>>>> *Tim Haverland*
>>>>> Acting Operations Branch Chief
>>>>> NOAA Fisheries Office of Science and Technology
>>>>> 1315 East-West Highway
>>>>> SSMC3 Rm 12303
>>>>> Silver Spring, MD 20910
>>>>> 301-427-8137 <tel:301-427-8137>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Open.ogc mailing list
>>>>> Open.ogc at list.woc.noaa.gov <mailto:Open.ogc at list.woc.noaa.gov>
>>>>> https://list.woc.noaa.gov/cgi-bin/mailman/listinfo/open.ogc
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Tim Haverland*
>>>> Acting Operations Branch Chief
>>>> NOAA Fisheries Office of Science and Technology
>>>> 1315 East-West Highway
>>>> SSMC3 Rm 12303
>>>> Silver Spring, MD 20910
>>>> 301-427-8137 <tel:301-427-8137>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://list.woc.noaa.gov/pipermail/open.ogc/attachments/20130924/cd4fc8cc/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 153411 bytes
Desc: not available
URL: <https://list.woc.noaa.gov/pipermail/open.ogc/attachments/20130924/cd4fc8cc/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 164721 bytes
Desc: not available
URL: <https://list.woc.noaa.gov/pipermail/open.ogc/attachments/20130924/cd4fc8cc/attachment-0003.png>
More information about the Open.ogc
mailing list